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INITIAL  DEVELOPMENT  AND  VALIDATION  OF  ASSESSMENTS  FOR 
PREDICTING  DISENROLLMENT  OF  FOUR- YEAR  SCHOLARSHIP  RECIPIENTS 
FROM  THE  RESERVE  OFFICER  TRAINING  CORPS 


EXECUTIVE  SUMMARY 


Research  Requirement 

The  Reserve  Officer  Training  Corps  (ROTC)  is  clearly  an  essential  commissioning 
source  for  the  U.S.  Army.  ROTC  has  recently  produced  the  majority  of  the  Army’s  new  Officers 
and  yields  Officers  that  eventually  reach  the  highest  ranks.  Analyses  have  indicated  that  Officers 
graduating  from  the  four- year  ROTC  scholarship  program  tend  to  be  more  likely  than  their  non¬ 
scholarship  ROTC,  U.S.  Military  Academy  (USMA),  and  Officer  Candidate  School  (OCS) 
counterparts  to  leave  after  their  initial  Active  Duty  Service  Obligation  (ADSO).  Furthermore, 
relative  to  non- scholarship  Cadets,  four- year  scholarship  Cadets  tend  to  be  less  likely  to 
complete  the  ROTC  program  and  become  commissioned  Officers.  Data  collected  for  the  current 
project  showed  that  approximately  10.3%  of  four-year  scholarship  freshmen  in  2007  disenrolled 
between  their  freshman  and  sophomore  years.  For  these  reasons,  the  U.S.  Army  Cadet  Command 
needs  valid,  reliable  measures  to  use  in  awarding  scholarships  to  individuals  who  are  likely  to 
complete  their  programs.  Toward  that  end,  the  U.S.  Army  Research  Institute  (ARI)  for  the 
Behavioral  and  Social  Sciences  initiated  a  research  project  to  develop  and  validate  an  instrument 
to  improve  selection  of  four- year  scholarship  recipients. 

The  short-term  goal  of  this  project  was  to  develop  a  new  measure  that  improves  the 
prediction  of  ROTC  continuance  for  four- year  scholarship  recipients,  beyond  the  level  of 
prediction  already  afforded  by  the  current  scholarship  award  process.  Specifically,  the  new 
measure  was  to  improve  prediction  beyond  that  provided  by  Whole  Person  Scores  (WPS)  that 
are  computed  based  on  the  applicant’s  responses  to  the  application.  The  long-term  objective  of 
this  project  is  to  provide  the  foundation  for  future  longitudinal  research  that  examines  the  impact 
of  the  new  measures  for  predicting  ROTC  program  completion,  commissioning,  and  career 
continuance  in  the  Army. 

Procedure 

ARI  developed  a  new  measure  to  improve  scholarship  recipient  selection — the  Cadet 
Background  and  Experiences  Form  (CBEF).  The  initial  version  of  the  CBEF  had  two  primary 
parts:  the  Rational  Biodata  Inventory  (RBI)  and  the  Propensity  for  Commitment  (PFC)  scales.  In 
addition  to  these  parts,  several  additional  items  were  included  to  help  evaluate  the  functioning  of 
the  RBI  and  PFC  scales.  In  the  spring  of  2007,  ARI  and  Cadet  Command  administered  the  CBEF 
to  freshman,  four- year  scholarship  recipients.  ARI  contracted  with  the  Human  Resources 
Research  Organization  (HumRRO)  to  (a)  analyze  data  arising  from  this  data  collection,  (b) 
evaluate  the  CBEF’s  functioning  and  potential  for  predicting  disenrollment,  and  (c)  make 
revisions  to  the  CBEF  to  improve  its  potential  for  identifying  applicants  at  heightened  risk  of 
subsequent  disenrollment. 
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HumRRO  collected  application  data  and  disenrollment  data  from  Cadet  Command  for 
four- year  scholarship  recipients  in  the  sample  and  created  an  analysis  database.  In  turn, 

HumRRO  analyzed  the  data  using  basic  statistical  approaches  and  logistic  regression. 

Findings 

Three  criterion  variables  were  used  to  evaluate  the  potential  of  the  CBEF’s  RBI  and  PFC 
scales  for  predicting  (a)  disenrollment  (versus  enrollment),  (b)  Cadets’  self-reported  likelihood  of 
becoming  an  Army  Officer,  and  (c)  Cadets’  self-reported  likelihood  of  making  the  Army  a 
career.  When  considered  in  isolation,  several  CBEF  scales  were  significantly  predictive  of  all 
three  criteria,  notably  RBI  Army  Identification,  Achievement  Orientation,  Fitness  Motivation, 
Self-Efficacy,  Hostility  to  Authority,  and  PFC  Persistence.  When  considered  in  the  context  of  all 
CBEF  scales  and  Whole  Person  Scores  (WPS)  via  logistic  regression  analyses:  (a)  only  the  RBI 
Army  Identification  scale  remained  significantly  related  to  all  three  criteria,  (b)  the  RBI 
Educational  Identification  scale  was  significantly  related  to  disenrollment  and  Cadets’  self-rated 
likelihood  of  becoming  an  Army  Officer,  and  (c)  the  RBI  Self-Efficacy  scale  was  significantly 
related  to  Cadets’  self-rated  likelihood  of  making  the  Army  a  career.  Furthermore,  these  logistic 
regression  analyses  indicated  that  addition  of  CBEF  scale  scores  (in  particular,  RBI  scale  scores) 
to  a  model  including  only  WPS  significantly  incremented  the  validity  of  WPS  for  predicting  all 
three  criteria. 

Utilization  and  Dissemination  of  Findings 

The  results  of  this  work  were  used  to  revise  the  CBEF.  Less  predictive  scales  were 
dropped  and  new  ones  were  added.  Also,  the  self-report  criteria  in  the  initial  study  proved  highly 
useful  and  have  now  been  expanded  in  a  new  self-report  criterion  measure  that  assesses  Cadets’ 
attitudes  towards  school,  ROTC,  and  the  Army.  ARI  and  Cadet  Command  are  planning  data 
collections  with  additional  cohorts  of  ROTC  four- year  scholarship  recipients.  In  the  end,  this 
project  will  provide  a  rich  source  of  data  for  studying  ROTC  program  completion  and  Army 
career  continuance  beyond  the  service  obligation  incurred  with  ROTC. 
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Initial  Development  and  Validation  of  Assessments  for  Predicting 
Disenrollment  of  Four-Year  Scholarship  Recipients  from  the 
Reserve  Officer  Training  Corps 

Chapter  1.  Overview 

Trueman  Tremble 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

Teresa  Russell 

Human  Resources  Research  Organization 

Background 

The  Reserve  Officer  Training  Corps  (ROTC)  is  clearly  an  essential  commissioning 
source  for  the  U.S.  Army.  ROTC  has  recently  produced  the  majority  of  the  Army’s  new 
Officers  (Wiedemann,  2005)  and  yields  Officers  that  eventually  reach  the  highest  ranks.  For 
example,  in  2005,  approximately  half  of  General  Officers  were  ROTC  graduates  while 
approximately  one-third  came  from  the  U.S.  Military  Academy  (USMA). 

The  Army  offers  several  avenues  to  Army  ROTC.  Qualified  high  school  seniors  or 
current  college  students  may  apply  for  two-,  three-,  or  four- year  scholarships.  In  each  case, 
students  must  meet  physical  and  academic  standards.  Enlisted  Soldiers  can  earn  a 
commission  as  an  Army  Officer  through  the  Army  Green  to  Gold  program.  This  program  has 
two  options — a  scholarship  option  for  Soldiers  who  are  considering  leaving  active  duty  to 
attend  college  and  an  active  duty  option  for  Soldiers  who  want  to  remain  on  active  duty  and 
attend  college.  Army  ROTC  students  who  receive  an  Army  ROTC  scholarship  must  agree  to 
complete  a  period  of  service  with  the  Army. 

Analyses  have  indicated  that  Officers  graduating  from  the  four- year  ROTC 
scholarship  program  tend  to  be  more  likely  than  their  USMA,  Officer  Candidate  School 
(OCS),  and  non-scholarship  ROTC  counterparts  to  leave  after  their  initial  Active  Duty 
Service  Obligation  (ADSO;  Doganca,  2006).  Furthermore,  relative  to  non-scholarship 
Cadets,  four-year  scholarship  Cadets  tend  to  be  less  likely  to  complete  the  ROTC  program 
and  become  commissioned  Officers.  Data  collected  for  the  current  project  showed  that 
approximately  10.3%  of  four-year  scholarship  freshmen  in  2007  disenrolled  between  their 
freshman  and  sophomore  years  (see  Chapter  3).  For  these  reasons,  the  U.S.  Army  Cadet 
Command  needs  a  valid,  reliable  measure  to  use  in  awarding  scholarships  to  individuals  who 
are  likely  to  complete  their  programs.  Toward  that  end,  the  U.S.  Army  Research  Institute 
(ARI)  for  the  Behavioral  and  Social  Sciences  initiated  a  research  project  to  develop  and 
validate  an  instrument  to  improve  selection  of  four- year  scholarship  recipients. 

The  Four-Year  High  School  Scholarship  Program  Application  and  Selection  Process 

Most  students  who  apply  for  a  four- year  scholarship  do  so  during  their  junior  or 
senior  year  of  high  school.  Each  applicant  who  qualifies  for  the  program  based  on  college 
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entrance  exam  scores,  high  school  grade  point  average  (HS  GPA),  and  date-of-birth 
requirements  is  interviewed  by  a  Professor  of  Military  Science  (PMS).  Offers  are  made  to 
applicants  on  the  basis  of  their  application  data,  interview  results,  and  an  overall  evaluation 
by  a  selection  board.  The  specific  steps  in  the  selection  process  are  listed  in  Figure  1.1. 


1.  A  high-school  student  completes  an  application  for  a  scholarship. 

2.  The  applicant  is  considered  “eligible,”  if  s/he  meets  three  criteria: 

■  Has  a  HS  GPA  of  2.5  or  greater 

-  Has  an  ACT>=  19  or  an  SAT>=920 

*  Meets  date-of-birth  eligibility 

3.  Every  few  weeks  after  this  “eligibility”  determination  is  made,  Cadet  Command  identifies  a  set  of 
eligible  applicants  to  interview.  Not  every  eligible  applicant  is  interviewed  at  once;  they  are 
interviewed  in  cycles. 

4.  Eligible  applicants  are  interviewed  by  the  Professor  of  Military  Science  (PMS)  at  the  school  closest  to 
the  home  of  record,  regardless  of  the  colleges  to  which  they  have  applied.  Interviews  may  be 
conducted  in  person  or  over  the  phone. 

5.  PMS  interviewers  complete  an  interview  form  and  return  the  form  to  Cadet  Command. 

6.  Once  a  sizable  number  of  interviews  are  completed  for  eligible  applicants.  Cadet  Command  holds 
selection  boards  to  determine  which  applicants  will  be  extended  scholarship  offers.  The  decision  to 
award  scholarship  is  based  on  the  sum  of  three  sets  of  points:  (a)  points  awarded  to  the  applicant  based 
on  the  PMS  interview,  (b)  points  awarded  to  the  applicant  based  on  “Whole  Person  Scores”  (WPS) 
derived  from  his/her  scholarship  application,  and  (c)  “board  points”  awarded  to  the  applicant  based  on 
the  selection  board’s  review  of  the  applicant’s  entire  package.1 

7.  Based  on  the  aforementioned  selection  board.  Cadet  Command  extends  offers  in  a  top  down  fashion 
based  on  total  points.  The  number  of  scholarships  awarded  in  any  given  promotion  board  cycle  varies, 
and  depends  on  myriad  factors  (e.g.,  quality  of  the  applicants,  slots  to  be  filled,  funds  remaining,  etc.). 
In  awarding  the  scholarship.  Cadet  Command  will  make  the  offer  good  for  up  to  three  schools  to  which 
the  applicant  had  applied  for  admission.  Limiting  the  choice  of  schools  serves  to  cap  the  amount  of 
scholarship  money  that  Cadet  Command  will  spend  on  an  applicant.  If  the  applicant  failed  to  get  into 
any  of  the  schools  on  the  list,  or  wishes  to  attend  another  school,  the  applicant  must  contact  Cadet 
Command  to  see  if  an  accommodation  can  be  made.  Applicants  who  do  not  receive  an  offer  in  the 
initial  selection  board,  are  reconsidered  by  subsequent  selection  boards  (though  their  total  points  are 
not  recalculated). 

8.  Once  the  offer  is  made,  the  applicant  has  30  days  to  accept  or  decline  the  offer.  If  the  applicant  does 
not  respond,  the  offer  is  withdrawn. 

9.  The  next  step  in  the  process  is  enrollment.  The  applicant  enrolls  in  the  ROTC  program  upon  arriving 
on  campus,  and  goes  through  steps  necessary  to  contract  with  the  Army  (e.g.,  undergo  a  medical  exam 
to  ensure  they  meet  minimum  physical  requirements,  and  background  check  to  ensure  they  meet  moral 
character  requirements). 

Figure  1.1.  Overview  of  the  four-year  high  school  scholarship  application-selection 
process. 


1  Whole  Person  Scores  (WPS)  reflect  points  awarded  to  applicants  based  on  the  four-year  high  school 
scholarship  application  form.  Points  are  awarded  for  (a)  college  board  scores  (i.e.,  SAT/ACT  scores),  (b)  high 
school  GPA,  and  (c)  extracurricular  activities  (e.g.,  participation  and/or  leadership  positions  held  in  student 
government,  sports,  clubs)  (Cadet  Command  Pamphlet  145-1). 
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Overview  of  Approach 


The  short-term  goal  of  this  project  was  to  develop  a  new  measure  that  improves  the 
prediction  of  ROTC  continuance  for  four- year  scholarship  recipients,  beyond  the  level  of 
prediction  already  afforded  by  the  current  scholarship  award  process.  The  long-term 
objective  of  this  project  is  to  provide  the  foundation  for  future  longitudinal  research  that 
examines  the  impact  of  the  new  measures  for  predicting  ROTC  program  completion, 
commissioning,  and  career  continuance  in  the  Army. 

With  those  objectives  in  mind,  ARI  developed  a  new  measure  to  improve  scholarship 
recipient  selection — the  Cadet  Background  and  Experiences  Form  (CBEF).  The  initial 
version  of  the  CBEF  had  two  primary  parts:  the  Rational  Biodata  Inventory  (RBI)  and  the 
Propensity  for  Commitment  (PFC)  scales.  In  addition,  early  indicators  of  ROTC  and  Army 
career  continuance  were  included  to  provide  a  preliminary  assessment  of  the  criterion-related 
validity  of  the  CBEF.  In  the  spring  of  2007,  ARI  and  Cadet  Command  administered  the 
CBEF  to  freshman,  four-year  scholarship  recipients.  ARI  contracted  with  the  Human 
Resources  Research  Organization  (HumRRO)  to  (a)  analyze  data  arising  from  this  data 
collection,  (b)  evaluate  the  CBEF’s  functioning  and  potential  for  predicting  disenrollment, 
and  (c)  make  revisions  to  the  CBEF  to  improve  its  potential  for  identifying  applicants  at 
heightened  risk  of  subsequent  disenrollment. 

Organization  of  the  Report 

Chapters  2  through  5  of  this  report  describe  the  initial  development  and  validation  of 
the  CBEF.  Specifically,  Chapter  2  discusses  the  development  of  the  CBEF.  Chapter  3 
discusses  the  administration  of  the  CBEF  to  four-year  scholarship  recipients  in  their 
freshman  year  of  Military  Science  (MS-1)  during  the  spring  of  2007,  as  well  as  the 
development  of  the  validation  database.  Chapter  4  discusses  the  basic  psychometric 
properties  of  the  CBEF  and  key  validation  criteria  used  in  this  effort  (e.g.,  disenrollment). 
Chapter  5  examines  criterion-related  validity  for  the  CBEF,  as  well  as  Whole  Person  Scores 
(WPS)  derived  from  the  four- year  scholarship  application.  Last,  Chapter  6  describes 
additional  efforts  underway  to  examine  the  psychometric  properties  and  functioning  of  the 
CBEF  and  future  research  plans. 
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Chapter  2.  Development  of  the  CBEF 

Robert  Kilcullen,  Jordan  Robbins,  &  Trueman  Tremble 
U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

In  an  effort  to  assist  the  Cadet  Command  with  its  initiative  to  reduce  disenrollment  of 
Army  ROTC  four-year  scholarship  recipients,  ARI  developed  the  Cadet  Background  and 
Experiences  Form  (CBEF).  The  CBEF  was  designed  as  a  tool  to  potentially  augment  the 
selection  criteria  currently  used  by  Cadet  Command  to  make  scholarship  decisions  (e.g., 
Whole  Person  Scores  based  on  the  four-year  scholarship  application,  PMS  interview  scores, 
selection  board  points).  The  purpose  of  this  chapter  is  to  describe  the  development  of  the 
CBEF. 


As  noted  in  Chapter  1,  the  CBEF  developed  for  the  current  effort  had  two  primary 
parts:  a  modified  version  of  the  Rational  Biodata  Inventory  (RBI)  for  use  with  ROTC  Cadets 
and  the  Propensity  for  Commitment  (PFC)  scales.  The  modified  RBI  measured  temperament 
constructs  hypothesized  to  relate  to  ROTC  disenrollment  and  Officer  continuance.  Part  II  of 
the  CBEF,  the  PFC,  consisted  of  41  contrasting  pairs  of  statements  designed  to  scale 
respondents’  propensity  for  commitment.  In  addition  to  the  RBI  and  PFC,  the  CBEF  also 
included  several  items  to  help  evaluate  the  functioning  of  the  RBI  and  PFC  scales.  These 
comprised  1 1  questions  that  asked  respondents  about  (a)  their  participation  in  Junior  ROTC 
(JROTC),  (b)  when  they  began  considering  enrolling  in  ROTC,  (c)  factors  that  influenced 
their  decision  to  join  ROTC,  and  (d)  their  self-rated  likelihood  of  completing  ROTC, 
becoming  a  commissioned  Officer,  and  making  the  Army  a  career. 

The  Rational  Biodata  Inventory  (RBI) 

The  RBI  was  included  in  the  CBEF  test  battery  because  it  has  been  a  consistent 
predictor  of  both  Soldier  job  performance  and  attrition  in  previous  research  (Kilcullen,  Putka, 
McCloy,  2007;  Klopp,  2006;  Putka  &  Bradley,  2008;  Putka,  Kilcullen,  &  White,  2003). 
Rational  biodata  scales  such  as  the  RBI  are  designed  to  measure  temperament  constructs 
using  questions  about  past  behavior  and  life  experiences.  Temperament  constructs  were 
targeted  for  measurement  with  the  original  version  of  the  RBI  based  on  a  job  analysis  of  the 
competencies  required  of  future  Soldiers  (Sager,  Russell,  R.C.  Campbell,  &  Ford,  2005),  as 
well  as  a  review  of  the  constructs  measured  by  other  biodata  tests  developed  by  ARI, 
particularly  the  Assessment  of  Right  Conduct  (ARC)  and  the  Test  of  Adaptable  Personality 
(TAP)  -  that  predict  both  counterproductive  behavior  and  job  performance  in  the  Army 
(Kilcullen,  Goodwin,  Chen,  Wisecarver,  &  Sanders,  2002;  Kilcullen,  Mael,  Goodwin,  & 
Zazanis,  1999;  Kilcullen,  White,  Sanders,  &  Hazlett,  2003).  A  detailed  description  of  the 
development  of  the  RBI  is  provided  by  Kilcullen,  Putka,  McCloy,  and  Van  Iddekinge  (2005). 

The  RBI  can  be  tailored  to  measure  different  temperament  constructs  of  interest  to  a 
given  population.  Two  new  RBI  scales  were  developed  for  use  with  ROTC  Cadets: 
Educational  Identification  and  Impulsiveness.  The  idea  behind  the  Educational  Identification 
scale  was  that  some  Cadet  disenrollment  could  be  due  to  Cadets  dropping  out  of  college, 
which  may  be  largely  motivational  (e.g.,  not  attending  class).  The  Impulsiveness  scale  was 
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added  because  some  Cadets  might  accept  the  scholarship  or  make  the  decision  to  leave 
without  thinking  through  the  decision.  The  RBI  administered  in  this  effort  (see  Figure  2.1) 
consisted  of  12  predictor  scales  and  a  Response  Distortion  scale  that  detected  whether  the 
respondents  were  describing  themselves  in  an  overly  favorable  light  (i.e.,  in  a  way  that  may 
not  reflect  their  true  standing  on  the  constructs  of  interest  on  the  RBI).  A  sample  RBI  item 
appears  in  Figure  2.2. 


Peer  Leadership:  Seeks  positions  of  authority  and  influence.  Comfortable  with  being  in  charge  of  a  group. 
Willing  to  make  tough  decisions  and  accept  responsibility  for  the  group’s  performance. 

Cognitive  Flexibility:  Willingness  to  entertain  new  approaches  to  solving  problems.  Enjoys  creating  new  plans 
and  ideas.  Initiates  and  accepts  change  and  innovation. 

Achievement  Orientation:  The  willingness  to  give  one’s  best  effort  and  to  work  hard  towards  achieving  difficult 
objectives. 

Fitness  Motivation:  Degree  of  enjoyment  from  participating  in  physical  exercise.  Willingness  to  put  in  the  time 
and  effort  to  maintain  good  physical  conditioning. 

Interpersonal  Skills  -  Diplomacy:  Being  extroverted  and  outgoing.  Able  to  make  friends  easily  and  establish 
rapport  with  strangers.  Good  at  meeting/greeting  people. 

Stress  Tolerance:  Ability  to  maintain  one’s  composure  under  pressure.  Remaining  calm  and  in  control  of  one’s 
emotions  instead  of  feeling  anxious  and  worried. 

Hostility  to  Authority:  Being  suspicious  of  the  motives  and  actions  of  legitimate  authority  figures.  Viewing  rules, 
regulations,  and  directives  from  higher  authority  as  punitive  and  illegitimate. 

Self-Efficacy:  Feeling  that  one  has  successfully  overcome  work  obstacles  in  the  past  and  that  one  will  continue  to 
do  so  in  the  future. 

Cultural  Tolerance:  Willingness  to  work  with  people  of  different  cultures.  Being  able  to  establish  supportive 
work  relationships  with  people  of  a  variety  of  racial  and  ethnic  backgrounds. 

Army  Identification:  The  degree  of  personal  identification  with,  and  intrinsic  interest  in  becoming,  a  U.S.  Army 
Soldier. 

Educational  Identification:  Desire  for  a  college  degree.  Perceives  self  as  someone  who  will  complete  college. 
Impulsiveness:  The  tendency  to  act  without  thinking  beforehand. 

Response  Distortion:  This  scale  is  not  a  predictor  scale.  Its  purpose  is  to  detect  and  adjust  for  socially  desirable 
responding.  (7  items) 

Figure  2.1.  ROTC  Rational  Biodata  Inventory  (RBI)  scales. 


How  often  have  you  put  off  doing  a  chore  that  you  could  have  taken  care  of  right  away? 

A.  Very  often 

B.  Often 

C.  Sometimes 

D.  Seldom 

_ E.  Never _ 

Figure  2.2.  Sample  RBI  item. 


Development  of  the  Propensity  for  Commitment  (PFC)  Scales 

Commitment  is  commonly  thought  to  be  a  psychological  state  that  represents  the 
nature  of  the  employees’  relationship  with  a  given  organization  and  that  influences  whether 
they  choose  to  continue  at  that  organization  (Meyer  &  Allen,  1991,  1997).  This 
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conceptualization  of  commitment  is  useful  in  understanding  the  extent  to  which  an  individual 
is  committed  to  an  organization  at  any  given  point  in  time,  but  it  offers  only  a  snapshot  of  an 
individual’s  current  commitment  state  and  says  little  about  an  individual’s  dominant 
tendencies  towards  organizational/group  continuance  behavior.  Furthermore,  Mowday, 
Porter,  and  Steers  (1982)  point  out  that  commitment  prior  to  organizational  entry  may  be 
meaningless  because  actual  commitment  to  remain  in  an  organization  exists  only  after  one 
has  entered  the  organization.  Thus,  while  commitment  remains  one  of  the  most  important 
predictors  of  retention,  it  may  not  be  well  suited  for  selection  contexts. 

Researchers  have  suggested  that  individuals  may  differ  in  the  extent  to  which  they  are 
dispositionally  inclined  to  become  committed  to  a  group  (Mowday  et  al.,  1982).  Although 
this  topic  has  received  relatively  little  attention  in  the  psychological  literature  (e.g.  Lee, 
Ashford,  Walsh,  &  Mowday,  1992;  Mowday  et  al.,  1982;  Wolf  &  Betz,  2004),  it  has  the 
potential  to  remedy  the  problems  of  using  commitment  in  selection  contexts.  For  example, 
individuals  can  have  the  tendency  to  orient  themselves  towards  a  group  or  persist  in  their 
group  memberships  even  prior  to  joining  a  given  group.  Expanding  on  this  area  of  research, 
ARI  began  in  a  prior  study  to  examine  an  individual’s  dominant  tendencies  towards 
organizational/group  continuance  behavior  as  a  potential  predictor  measure  that  could  be 
administered  to  applicants  prior  to  organizational  entry.  More  specifically,  ARI  raised  the 
possibility  that  people  differ  with  respect  to  their  base-level  propensity  for  commitment 
(PFC).  ARI  proposed  further  that  individuals  stronger  in  their  base-level  PFC  and  voluntarily 
entering  an  organization  are  more  likely  to  become  committed  to  and  remain  in  the 
organization. 

Propensity  for  commitment  (PFC)  was  conceptualized  as  an  individual’s  generalized 
tendency  to  become  committed  to  a  group  or  organization.  Individuals  high  in  PFC  should 
strive  to  seek  out  social  bonds  or  relationships,  socially  immersing  and  integrating 
themselves  into  a  group  in  order  to  feel  as  if  they  are  part  of  something  larger  than 
themselves,  and  to  persist  in  these  relationships  once  they  are  formed. 

Initial  Item  Development  and  Pilot  Test 

The  PFC  is  an  experimental  measure  developed  in  an  unpublished  pilot  test 
conducted  by  ARI  prior  to  the  study  described  in  this  report.  The  initial  scale  consisted  of 
thirty-six  bipolar  items,  designed  to  tap  individuals’  generalized  values,  reactions,  and 
justifications  regarding  their  tendency  to:  (a)  seek  out  others  and  work  well  as  a  group 
member,  (b)  identify  with  and  internalize  their  membership  in  a  group,  and  (c)  persist  in 
group  membership.  The  prior  study  piloted  the  PFC  on  294  junior  Officers  taking  part  in  the 
Basic  Officer  Leadership  Course  at  Fort  Benning.  Confirmatory  factor  analysis  indicated  that 
the  measure  was  composed  of  two  factors  that  were  labeled  Persistence  and  Group 
Orientation.  The  Group  Orientation  dimension  refers  to  an  individual’s  tendency  to  seek  out 
and  affiliate  with  a  group,  while  the  Persistence  dimension  refers  to  a  tendency  to  persist  in 
one’s  group  membership  and  to  see  oneself  as  a  group  member  regardless  of  the 
circumstances.  The  internal  consistency  (inter-item)  reliabilities  for  these  scales  were 
marginal,  having  coefficient  alphas  between  .57  and  .62.  Nevertheless,  the  scales  seemed  to 
have  modest  convergent  and  discriminant  validity  with  other  measures  such  as  Need  for 
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Belonging,  Need  for  Identification,  Organizational  Commitment,  Intent  to  Remain  in  the 
Army,  and  the  Big  Five  personality  traits  (Openness  to  Experience,  Conscientiousness, 
Extraversion,  Agreeableness,  and  Neuroticism).  In  sum,  the  relatively  low  reliabilities 
suggested  that  further  work  would  be  needed  to  ensure  that  the  scales  are  tapping  the 
constructs  of  interests. 

Based  on  results  and  feedback  received  from  the  prior  pilot  test,  ARI  took  steps  to 
improve  the  PFC  for  administration  in  the  spring  of  2007.  For  example,  instructions  were 
modified  to  clarify  potential  points  of  confusion  regarding  the  use  of  the  scales.  Items  were 
modified  that  seemed  to  be  possibly  confusing,  and  new  items  were  added.  The  revised  PFC 
consisted  of  41  pairs  of  statements.  One  statement  in  each  pair  was  hypothesized  to  reflect 
low  standing  on  the  target  construct  of  interest  (e.g.,  Group  Orientation),  and  the  other 
statement  in  the  pair  was  hypothesized  to  reflect  high  standing  on  the  target  construct  of 
interest.  As  shown  in  Figure  2.3,  respondents  were  asked  to  respond  to  each  pair  of 
statements  using  a  semantic  differential-like  scale  (Snider  &  Osgood,  1969). 


The  boxes  in  each  row  below  present  you  with  two  opposing  statements.  Mark  each  row  in  the  way 
that  best  describes  you.  For  example,  assume  you  were  presented  with  the  following: 

I  feel  happy  ()()()()()  I  feel  sad 

If  you  are  typically  a  very  happy  person  you  would  mark  your  answer  on  the  scale  as  follows: 

I  feel  happy  (X)  ()()()()  I  feel  sad 

If,  on  the  hand,  you  are  typically  feel  somewhat  sad,  you  would  mark  your  answer  as: 

I  feel  happy  ()()()  (X)  ( )  I  feel  sad 

Please  be  aware  that  the  contrasting  statements  will  be  different  for  each  question.  The  closer  a  mark 
is  to  a  given  statement,  the  greater  it  indicates  your  agreement  with  that  statement.  Please  read  both 
statements  on  a  given  row  before  indicating  your  response. 


Figure  2.3.  PFC  instructions. 

Additional  Items  on  the  CBEF 

As  noted  earlier,  in  addition  to  the  RBI  and  PFC  scales,  the  CBEF  also  included 
several  items  to  help  evaluate  the  functioning  of  said  scales.  These  items  asked  respondents 
about: 

•  Participation  in  the  junior  ROTC  program. 

•  Importance  of  various  factors  (e.g.,  pay  for  college,  please  Family)  in  making  the 
decision  to  enroll  in  ROTC. 

•  Fikelihood  that  the  Cadet  will  complete  ROTC  and  become  a  commissioned  Officer. 

•  Fikelihood  that  the  Cadet  will  make  the  Army  a  career. 

As  described  in  Chapter  4,  Cadets’  responses  to  the  latter  two  items,  along  with  their 
enrollment  status  (obtained  from  archival  Cadet  Command  records)  served  as  preliminary 
criteria  when  evaluating  the  RBI  and  PFC  scales.  Specifically,  these  were  administered  to 
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Cadets  who  completed  the  CBEF  so  we  could  examine  whether  RBI  and  PFC  scales  could 
predict  early  indicators  of  ROTC  and  Army  career  continuance. 
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Chapter  3.  Administration  of  the  CBEF  and  Development  of  the  Validation  Database 


Trueman  Tremble  and  Robert  Kilcullen 
U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

Ani  DiFazio  and  Dan  J.  Putka 
Human  Resources  Research  Organization 

In  the  spring  of  2007,  over  1,900  copies  of  the  Cadet  Background  and  Experiences 
Form  (CBEF)  along  with  administration  instructions  were  distributed  to  Professors  of 
Military  Science  (PMSs)  at  over  250  colleges  across  the  country.  The  goal  of  distributing 
these  packets  was  to  have  PMSs  administer  the  CBEF  to  all  four-year  high  school 
scholarship  recipients  enrolled  in  their  freshman  year  of  military  science  (MS-1).  This 
chapter  discusses  the  process  of  administering  the  CBEF  to  these  Cadets,  as  well  as 
subsequent  construction  of  a  database  designed  to  support  the  validation  of  the  measure.  This 
database  reflected  not  only  data  collected  from  the  aforementioned  Cadets  during  the  spring 
2007  administration  of  the  CBEF,  but  also  archival  data  from  Cadet  Command  that  contained 
these  Cadets’  application  data  (e.g.,  their  Whole  Person  Scores),  as  well  as  their  enrollment 
status  as  of  the  fall  of  2007. 


Administration  of  the  CBEF 

The  PMSs  played  a  critical  role  in  administering  the  CBEF.  Along  with  the  copies  of 
the  CBEF  described  above,  PMSs  were  provided  with  rosters  of  Cadets  believed  to  (a)  have 
four- year  scholarships,  and  (b)  be  in  the  their  freshman  year  of  college.  In  preparation  for  the 
testing  sessions,  PMSs  were  asked  to  review  and  update  a  roster  of  four- year  freshman 
scholarship  recipients,  become  familiar  with  the  study  and  materials,  and  make  arrangements 
for  testing."  The  PMSs  then  administered  the  CBEF  to  the  MS-1  Cadets  in  their  jurisdiction. 
For  future  reference,  the  specific  instructions  PMSs  were  given  for  preparing  for  and 
conducting  the  data  collection  sessions  are  presented  in  the  appendix  of  this  report. 

Development  of  the  Validation  Database 

Data  from  the  CBEF  administration  were  key-entered  and  merged  with  archival  data 
from  Cadet  Command  to  form  the  validation  database.  This  section  provides  details  of  those 
steps. 

The  CBEF  Dataset 

As  mentioned,  the  CBEF  was  administered  to  freshman  Cadets  in  paper- and-pencil 
form.  ARI  key-entered  the  raw  data  and  provided  electronic  CBEF  data  to  HumRRO. 
HumRRO  assessed  the  accuracy  of  ARI  key-entry  by  re -keying  the  151  variables  in  the 
CBEF  for  150  respondents  randomly  selected  by  ARI.  HumRRO  project  staff  then 
electronically  compared  our  key  entry  to  that  of  ARI  and  examined  hard  copies  of  surveys  in 


2  Rosters  provided  to  each  PMS  were  based  on  data  provided  by  Cadet  Command. 
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all  instances  where  variable  values  differed  between  ARI  and  HumRRO  key-entry.  Results  of 
this  effort  indicated  an  ARI  key  entry  error  rate  of  less  than  one  half  of  one  percent.  Satisfied 
with  the  level  of  accuracy  in  ARI  data  entry,  HumRRO  edited  CBEF  data  to  correct  the  key 
entry  errors  discovered  during  the  aforementioned  assessment  of  ARI  data  entry  and  created 
an  analysis  CBEF  dataset  comprising  data  for  1,571  Cadets  from  225  schools.  Given  that 
approximately  1,900  questionnaires  were  originally  distributed  across  256  schools,  this 
represents  an  individual-level  response  rate  of  approximately  82.7%  and  a  school-level 
response  rate  of  87.9%.  One  should  take  caution,  however,  when  interpreting  these  response 
rates  as  PMSs  at  each  school  had  the  discretion  to  prune  back  or  add  to  the  initial  roster  of 
Cadet’s  targeted  for  questionnaire  distribution.  For  example,  PMSs  at  a  given  school  would 
have  eliminated  a  Cadet’s  name  from  the  distribution  list  if  that  Cadet  was  no  longer  enrolled 
in  the  ROTC  program  as  of  the  spring  of  2007. 

Cadet  Command  Data 

Archival  data  on  Cadets  was  extracted  from  two  databases  maintained  by  Cadet 
Command:  the  Applicant  Database  and  the  Student  Management  Database.  The  Applicant 
Database  contains  myriad  data,  including  data  from  the  ROTC  four- year  high  school 
scholarship  application.  This  includes  a  variety  of  information  from  the  applicant,  such  as 
high  school  grades,  SAT/ACT  scores,  and  participation  in  high  school  activities.  This 
information  is  used  to  create  a  number  of  composites,  and  ultimately,  “Whole  Person  Scores” 
for  each  applicant,  which  play  a  key  role  in  the  scholarship  awarding  process  (see  Chapter  1). 
In  August  of  2007,  Cadet  Command  provided  electronic  data  from  the  Applicant  Database 
for  all  four- year  high  school  scholarship  Cadets  who  first  enrolled  during  the  2006-2007 
school  year  (i.e.,  SY06-07).  This  file  included  information  on  applicant  demographics,  high 
school  courses  and  standing,  test  scores,  academic  and  ROTC  school  preferences,  academic 
major,  composite  scores  and  a  code  that  indicates  the  status  of  the  applicant’s  scholarship 
(i.e.,  accepted,  declined,  or  withdrawn).  It  contained  records  describing  9,027  individual 
applicants.  In  the  final  dataset,  we  retained  all  “accepted”  applicants —  the  1,695  applicants 
coded  in  the  Application  Database  as  having  accepted  a  four- year  high  school  scholarship. 

The  second  database,  the  Student  Management  Database,  tracks  students  once  they 
enroll  in  ROTC  courses  in  college.  It  contains  data  required  for  the  student  to  contract  with 
the  Army  in  the  final  step  of  the  admittance  process  (e.g.,  age,  dependents)  and  key  variables, 
such  as  enrollment  status,  disenrollment  reason  (if  disenrolled),  Army  Physical  Fitness  Test 
(APFT)  scores,  and  grade  point  average  (GPA).  In  October  2007,  Cadet  Command  provided 
an  extract  from  the  Student  Management  Database  that  contained  the  latest  status  for  four- 
year  high  school  scholarship  Cadets  who  first  enrolled  during  SY06-07. 

We  merged  the  student  management  data  with  applicant  data  to  create  the  final  Cadet 
Command  dataset.  It  is  important  to  note  that  the  two  Cadet  Command  databases  were 
developed  independently  and  serve  different  purposes.  Consequently,  the  transition  between 
the  two  is  not  simple.  For  example,  only  those  applicants  who  accept  an  offer  (as  indicated  in 
the  Applicant  Database)  and  then  actually  enroll  for  their  freshman  year  actually  appear  in 
the  Student  Management  Database.  The  final  merged  file  included  data  from  the  two  merged 
files  on  all  1,695  cases  that  had  been  coded  as  accepting  a  scholarship. 
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The  Validation  Database 


The  CBEF  dataset  and  the  Cadet  Command  dataset  were  merged  to  produce  a  datafile 
to  support  all  analyses  (i.e.,  the  validation  database).  The  resulting  validation  database 
comprised  records  for  1,068  Cadets.  Recall  that  CBEF  data  were  available  for  1,571  Cadets, 
and  1,695  Cadets  had  accepted  four- year  high  school  scholarship  offers  according  to  Cadet 
Command’s  Applicant  Database.  Merging  these  datasets  revealed  that  503  (32%)  of  Cadets 
with  CBEF  data  did  not  appear  in  Cadet  Command’s  database  as  four-year  high  school 
scholarship  recipients.  When  this  finding  was  shared  with  Cadet  Command,  they  indicated 
that  it  was  within  reason  that  about  one-third  of  Cadets  who  completed  the  CBEF  in  the 
spring  of  2007  were  part  of  a  program  (on-campus  or  Green-to-Gold)  other  than  the  high 
school  scholarship  program.  As  such,  we  restricted  the  validation  database  used  here  to  those 
Cadets  who  (a)  completed  the  CBEF  as  part  of  the  spring  of  2007  data  collection  and  (b) 
accepted  four- year  high  school  scholarship  offers  based  on  the  SY06-07  application  data. 

Table  3.1  presents  basic  demographic  information  for  full  set  of  1,068  Cadets 
comprising  the  validation  database.  As  Table  3.1  shows,  the  majority  of  the  sample  was  male 
(78.2%)  and  White  (85.3%).  Comparison  of  these  numbers  to  characteristics  of  applicants 
who  accepted  offers  for  four- year  high  school  scholarships  for  SY06-07  (based  on  Cadet 
Command  records)  revealed  a  great  deal  of  similarity.  For  78.1%  of  applicants  who  accepted 
four-year  high  school  scholarships  for  SY06-07  were  male,  82.7%  were  White,  and  5.9% 
were  Black. 

With  the  validation  database  constructed,  we  next  screened  the  CBEF  data  for 
missing  data  (i.e.,  no  more  than  10%  missing  on  either  the  RBI  or  PFC),  as  well  as  random 
responding.  This  screening  eliminated  CBEF  data  for  34  of  1,068  Cadets  (3.4%)  in  the 
validation  database.  A  final  screen  was  done  to  ensure  that  individuals  who  disenrolled  prior 
to  the  administration  of  the  CBEF  were  excluded  from  analyses  (resulting  in  the  elimination 
of  two  additional  cases). 
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Table  3.1.  Demographic  Composition  of  Cadets  in  the  Validation  Database 


Variable 

Category 

N 

% 

Gender 

Female 

233 

21.82 

Male 

835 

78.18 

Race 

White 

911 

85.30 

Asian 

84 

7.87 

Hispanic 

68 

6.37 

Black 

55 

5.15 

Other 

33 

3.09 

American  Indian 

20 

1.87 

Hawaiian 

16 

1.50 

Note.  Applicants  could  designate  more  than  one 
race. 
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Chapter  4.  Basic  Psychometric  Properties  of  the  CBEF  and  Key  Criterion  Measures 

Shonna  Waters  and  Gordon  Waugh 
HumRRO 

This  chapter  describes  basic  psychometric  properties  of  the  CBEF,  based  on  the 
sample  of  Cadets  who  completed  the  measure  in  the  Spring  of  2007.  Additionally,  we 
describe  the  standing  of  these  Cadets  on  key  criteria  that  will  provide  the  basis  of  criterion- 
related  validity  analyses  described  in  Chapter  5. 

Scoring  of  the  RBI  and  CBEF  Components  of  the  CBEF 

Prior  to  performing  any  analyses  on  the  RBI  and  PFC,  we  scored  Cadets’  responses  to 
each  of  the  items  on  these  components  of  the  CBEF  and  aggregated  item  scores  into  scale 
scores  (described  in  Chapter  2).  The  scoring  of  the  RBI  was  straightforward.  Cadets 
responded  to  each  item  using  a  5-point  scale  (ranging  from  1  to  5),  and  items  pertinent  to 
each  RBI  scale  (see  Figure  2.2)  were  reverse  coded  as  needed  and  averaged  together  to  form 
RBI  scale  scores. 

The  scoring  of  the  PFC  was  not  quite  as  straightforward,  due  in  part  to  the  novelty  of 
the  measure,  and  nature  of  how  its  items  were  scaled.  Therefore,  we  conducted  several 
analyses  to  refine  the  PFC  item  pool  and  scale  the  items. 

Initially,  we  conducted  a  parallel  analysis  to  estimate  the  number  of  factors  needed  to 
explain  the  variance  in  the  PFC  items.  These  analyses  were  conducted  among  all  Cadets  who 
completed  at  least  90%  of  the  PFC  items  as  part  of  the  Spring  2007  data  collection  (N  = 
1,550).  We  found  that  about  15  factors  would  be  needed  to  explain  all  of  the  variance  in 
items. 1  Even  so,  we  found  that  the  first  two  or  three  factors  accounted  for  the  majority  of  the 
common  variance  (i.e.,  the  first  two  factors  account  for  57%  of  this  variance,  and  the  first 
three  factors  account  for  67%). 

Next,  we  used  a  series  of  Rasch  analyses  (1 -parameter  IRT  model)  to  determine 
whether  the  PFC  should  be  scored  on  one  scale  (e.g.,  total  score)  or  more  than  one  scale  (e.g., 
Persistence  and  Orientation)  and  found  that  more  than  one  scale  was  needed.  In  the  first 
Rasch  analysis,  we  included  all  41  items.  Based  on  this  initial  analysis,  Item  9  was  eliminated 
because  it  had  extremely  poor  statistics  (e.g.,  it  had  a  negative  item-total  correlation).  Other 
items  fit  poorly,  though  not  so  poorly  that  they  warranted  exclusion  from  subsequent 
diagnostic  analyses.  Six  persons  were  also  dropped  based  on  this  analysis  because  their 
responses  were  very  inconsistent  with  predicted  Rasch  model  values.  After  dropping  this 


3  A  parallel  analysis  compares  the  research  data  with  random  data  to  estimate  the  number  of  factors  underlying 
the  data.  The  factor  analyses  used  multiple-squared  correlations  as  the  initial  communality  estimates.  The 
iterated  principal  factor  method  was  used  to  extract  the  factors.  The  scree  plot  (i.e.,  plot  of  each  eigenvalue  vs. 
its  factor  number)  from  the  research  data  was  compared  with  the  scree  plots  of  100  random  samples  possessing 
the  same  number  of  items  (41)  and  the  same  sample  size  (Listwise  N  =  1460)  as  the  research  sample.  For  each 
research/random  pair,  the  number  of  factors  indicated  by  the  parallel  analysis  is  the  factor  number  just  before 
the  two  scree  plots  cross.  Among  the  100  random  samples,  37  indicated  14  factors  and  63  indicated  15  factors. 
Thus,  about  15  factors  were  needed  to  recover  the  common  variance  present  in  the  items. 
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item  and  these  persons,  40  items  and  1,544  persons  remained.  A  principal  components 
analysis  of  the  residuals  revealed  that  the  Rasch  (first)  factor  accounted  for  only  37%  of  the 
measured  variance,  and  there  here  were  two  meaningful  residual  factors  that  accounted  for 
6.9%  and  6.0%  of  the  unexplained  variance,  respectively.  This  pattern  of  findings  indicates 
that  a  single  scale  for  the  40  items  would  be  inappropriate. 

Next,  we  conducted  separate  Rasch  analyses  to  assess  and  refine  two  factors  that 
appeared  to  account  for  covariance  among  PFC  items — Persistence  and  Group  Orientation.4 
After  several  iterations  of  identifying  the  Persistence  items  with  the  worst  fit  (for  a  single 
Persistence  factor  solution),  nine  items  were  dropped  because  of  poor  fit  statistics.  The 
estimated  reliability  of  the  remaining  10  items  was  just  slightly  lower  (.68)  than  for  the 
original  19  items  (.70).  In  the  same  way,  nearly  half  of  the  items  in  the  Group  Orientation 
scale  were  dropped.  The  scale  decreased  from  21  items  to  12  items,  but  its  estimated 
reliability  dropped  only  slightly,  from  .79  to  .78. 

In  sum,  the  revised  version  of  the  PFC  Persistence  and  Group  Orientation  scales  was 
shortened  considerably,  but  retained  most  of  the  reliability  of  the  original  length  PFC.  The 
final  scores  for  the  PFC  were  computed  using  only  the  final  22  items.  Rasch  scores  and 
summated  scores  differ  in  two  respects.  First,  Rasch  scores  are  typically  scaled  so  that  most 
of  the  scores  are  between  -3  and  +3.  Second,  a  Rasch  scale  tends  to  be  stretched  out  near  the 
upper  and  lower  ends.  Thus,  the  relationship  between  the  summated  PFC  scores  and  the 
Rasch  scores  is  nearly  linear  except  at  the  low  end  and  the  high  end.  Before  analyzing  the 
data,  the  Rasch  scores  were  rescaled  so  that  they  were  as  similar  as  possible  (in  terms  of  their 
means  and  standard  deviations)  to  the  summated  scores.  For  example,  the  rescaled  Rasch  and 
summated  scores  have  the  same  means  and  very  similar  standard  deviations.  The  Rasch 
scores  were  correlated  .97  with  the  summated  scores.  Thus,  the  results  of  the  analyses  would 
have  been  only  trivially  different  if  summated  scores  had  been  used.  Nevertheless,  for  this 
effort  we  adopted  the  Rasch  scoring,  as  it  will  tend  to  make  more  of  a  difference  in  the  upper 
and  lower  ends  of  the  score  distribution,  where  selection  decisions  (both  screen  in,  and  screen 
out)  tend  to  be  made.  The  PFC  scales  reported  in  the  remaining  sections  of  this  report  were 
scored  as  are  as  follows: 

Rescaled  PFC  Persistence  =  33.78261  +  8.695652  *  Rasch  Score  (1) 

Rescaled  PFC  Group  Orientation  =  36.42857  +  9.52381  *  Rasch  Score  (2) 


Basic  Properties  of  the  RBI  and  PFC  Components  of  the  CBEF 
RBI  and  PFC  Scale  Statistics 

RBI  and  PFC  scale  means,  standard  deviations,  and  internal  consistency  (inter-item) 
reliabilities  appear  in  Table  4.1.  In  general,  the  RBI  and  PFC  scales  show  acceptable  variance 


4  The  decision  to  refine  a  two  factor  solution  as  opposed  to  a  three  factor  solution  was  based  on  the  fact  that  (a) 
two  factors  were  found  two  underlie  PFC  items  during  its  initial  pilot  testing  (see  Chapter  2),  and  (b)  the 
evidence  for  the  existence  of  a  meaningful  third  factor  was  not  deemed  strong  enough  to  warrant  its  addition. 
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and  reliability.  An  exception  was  the  Educational  Identification  scale  (EDID),  which  showed 
relatively  low  reliability  (a  =  .49),  indicating  that  further  work  is  needed  on  the  development 
of  this  scale.  We  should  note  that  although  the  Response  Distortion  scale  items  exhibited 
little  internal  consistency,  this  was  expected.  The  items  on  this  scale  were  not  hypothesized 
to  be  reflective  of  a  “response  distortion”  construct,  but  rather  formative  indicators  of  it 
(MacKenzie,  Podsakoff,  &  Jarvis,  2005).  As  noted  by  previous  researchers,  notions  of  inter¬ 
item  consistency  for  such  formative  measures  have  little  meaning  (Bollen  &  Lennox, 

1991) — we  report  a  reliability  for  the  Response  Distortion  scale  here  only  for  purposes  of 
contrasting  results  with  the  other  RBI  scale. 

Table  4.1  also  shows  correlations  among  the  CBEF  scales.  On  average,  the  RBI 
scales  were  minimally  to  moderately  correlated  with  (a)  each  other  ( Mean  r  =  .15,  Min  r  = 
-.30,  Max  r  =  .54),  and  (b)  the  two  PFC  scales  ( Mean  r  =  .14,  Min  r  -.12,  Max  r  =  .40).  The 
strongest  correlations  among  RBI  scales  were  among  the  Peer  Leadership,  Achievement 
Orientation,  Self-Efficacy,  and  Educational  Identification  scales  (all  were  positively 
correlated).  The  strongest  correlations  between  RBI  and  PFC  scales  were  for:  RBI 
Interpersonal  Skill  and  PFC  Group  Orientation  (r  =  .40),  RBI  Achievement  and  RBI  Army 
Identification  with  PFC  Persistence  (r’s  =  .30  and  .31  respectively).  The  pattern  of 
correlations  described  above  is  consistent  with  a  priori  expectations  and  helps  reinforce 
claims  that  these  measures  are  assessing  the  constructs  they  were  designed  to  measure. 

Lastly,  the  correlation  between  the  two  PFC  scales  was  .39 — suggesting  that  Persistence  and 
Group  Orientation  were  tapping  distinct  elements  of  the  propensity  for  commitment. 

In  addition  to  examining  relationships  among  CBEF  scales,  we  also  examined  their 
relationship  to  Whole  Person  Scores  (WPS)  derived  from  the  four-year  scholarship 
application  form.  Recall,  that  the  WPS  play  a  key  role  in  the  awarding  of  scholarships.  Thus, 
it  is  important  to  examine  the  relation  between  CBEF  scales  and  WPS  to  determine  potential 
redundancy  of  the  CBEF  scales.  If  CBEF  scales  are  redundant  with  WPS,  their  utility  for 
complementing  WPS  as  a  potential  predictor  of  disenrollment  would  be  limited.  As  shown  in 
Table  4.2,  there  was  very  little  overlap  between  the  WPS  and  the  CBEF  scales,  based  on  raw 
correlations  (r  =  .01  to  .15).  It  is  important  to  note  that  the  low  raw  correlation  between  the 
CBEF  scales  and  WPS  was,  in  part,  due  to  the  range  restriction  on  the  WPS.  For  example, 
the  average  WPS  among  the  sample  of  freshmen  completing  the  CBEF  was  237.58,  with  a 
standard  deviation  of  40.22,  whereas  the  average  WPS  among  all  SY06-07  applicants  was 
152.62  with  a  standard  deviation  of  107.24.  As  such,  the  sample  being  used  in  these  analyses 
was  subject  to  a  substantial  amount  of  range  restriction.  As  Table  4.2  reveals,  when  the  range 
restriction  in  WPS  was  accounted  for,  the  CBEF-WPS  correlations  increased  dramatically 
but  not  to  the  level  that  would  suggest  that  any  given  CBEF  scale  was  redundant  with  the 
WPS. 
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Table  4.1.  RBI  and  PFC  Scale  Means,  Standard  Deviations,  Reliabilities,  and  Intercorrelations 


Scale 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

1.  RBI:  Peer  Leadership  (PLEAD) 

.72 

2.  RBI:  Cognitive  Flexibility  (CF) 

.45 

.77 

3.  RBI:  Achievement  Orientation  (ACH) 

.47 

.40 

.71 

4.  RBI:  Fitness  Motivation  (FM) 

.29 

.12 

.25 

.77 

5.  RBI:  Interpersonal  Skills  -  Diplomacy  (ISDIP) 

.43 

.25 

.32 

.17 

.80 

6.  RBI:  Stress  Tolerance  (ST) 

.12 

.08 

.00 

.22 

.22 

.68 

7.  RBI:  Hostility  to  Authority  (HOST) 

-.01 

-.09 

-.21 

.03 

-.04 

-.30 

.61 

8.  RBI:  Self-Efficacy  (SE) 

.54 

.42 

.54 

.45 

.35 

.30 

-.13 

.77 

9.  RBI:  Cultural  Tolerance  (CT) 

.29 

.40 

.24 

.11 

.33 

.18 

-.21 

.32 

.74 

10.  RBI:  Army  Identification  (AI) 

.26 

.11 

.20 

.32 

.15 

.21 

-.12 

.31 

.14 

.78 

11.  RBI:  Impulsiveness  (IMP) 

.07 

-.01 

-.12 

.06 

.09 

-.21 

.43 

.00 

-.04 

-.09 

.62 

12.  RBI:  Educational  Identification  (EDID) 

.26 

.21 

.54 

.08 

.26 

.03 

-.28 

.32 

.17 

.09 

-.13 

.49 

13.  RBI:  Response  Distortion  (RD) 

.13 

.15 

.22 

.10 

.12 

.18 

-.24 

.20 

.14 

.08 

-.14 

.12 

.14 

14.  PFC:  Persistence  (PERSIST) 

.27 

.15 

.30 

.20 

.10 

.09 

-.06 

.29 

.08 

.31 

-.12 

.17 

.08 

.63 

15.  PFC:  Group  Orientation  (GRORIENT) 

.15 

.08 

.20 

.11 

.40 

.12 

.01 

.17 

.16 

.15 

.06 

.13 

.05 

.39 

.75 

Mean 

3.80 

3.64 

3.90 

3.84 

3.89 

3.08 

1.92 

4.21 

3.91 

4.00 

3.08 

4.42 

0.07 

39.76 

40.66 

SD 

0.53 

0.62 

0.47 

0.60 

0.69 

0.46 

0.45 

0.44 

0.62 

0.63 

0.50 

0.48 

0.12 

5.62 

6.97 

No.  of  Items 

6 

7 

9 

7 

5 

11 

7 

6 

5 

7 

6 

5 

2 

11 

12 

Note:  N  =  1030  -  1034.  The  response  distortion  scale  includes  7  items;  however,  5  items  had  no  variance.  Reliabilities  indexing  consistency  among  scale  items 
(via  coefficient  alpha)  are  on  the  diagonal.  Bold  indicates  correlation  was  significant  (p  <  .05,  one-tailed). 
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Table  4.2.  Correlations  between  the  WPS  and  RBI  and  PFC  Scales 


CBEF  Scale 

Correlation  with  WPS 

r  rc 

RBI:  Peer  Leadership  (PLEAD) 

.09 

.24 

RBI:  Cognitive  Flexibility  (CF) 

.11 

.29 

RBI:  Achievement  Orientation  (ACH) 

.08 

.21 

RBI:  Fitness  Motivation  (FM) 

.04 

.10 

RBI:  Interpersonal  Skills  -  Diplomacy  (ISDIP) 

.00 

.00 

RBI:  Stress  Tolerance  (ST) 

.07 

.19 

RBI:  Hostility  to  Authority  (HOST) 

-.05 

-.13 

RBI:  Self-Efficacy  (SE) 

.06 

.15 

RBI:  Cultural  Tolerance  (CT) 

.05 

.13 

RBI:  Army  Identification  (AI) 

-.04 

-.09 

RBI:  Impulsiveness  (IMP) 

-.07 

-.18 

RBI:  Educational  Identification  (EDID) 

.16 

.39 

RBI:  Response  Distortion  (RD) 

-.10 

-.25 

PFC:  Persistence  (PERSIST) 

.03 

.08 

PFC:  Group  Orientation  (GRORIENT) 

-.09 

-.23 

Note.  N  =  1030-1034.  r  =  Raw  correlation.  rc=  Correlation  corrected  for  direct  range  restriction  on  the  WPS 
(Case  2;  Thorndike,  1949).  Boldface  text  indicates  raw  correlation  was  significant  (p  <  .05,  one-tailed). 

Subgroup  Differences 

A  particularly  important  property  of  any  measure  being  considered  to  select 
scholarship  winners  is  whether  scores  on  the  measure  exhibit  mean  differences  across 
subgroups  (e.g.,  do  males  perform  differently  from  females?  do  Black  applicants  perform 
differently  from  White  applicants?).  To  the  extent  that  sizable  differences  exist,  the  measure 
may  be  perceived  as  unfair;  thus,  we  examined  the  possibility  of  subgroup  differences  on  the 
CBEF  and  WPS  here. 

Subgroup  differences  are  described  in  terms  of  Cohen’s  d  effect  sizes  (Cohen,  1988). 
Cohen  d  reflects  the  mean  difference  in  scores  on  a  given  measure  (e.g.,  the  RBI)  across 
subgroups  (e.g.,  males,  females)  and  is  expressed  in  standard  deviation  units.  Cohen  (1988) 
suggests  that  <r/’s  of  .20  in  magnitude  represent  a  small  difference  between  groups,  .50  a 
moderate  difference,  and  .80  a  large  difference.  The  formula  for  Cohen’s  d  used  in  the 
sections  that  follow  is: 


d  —  (T^Non-referent  group  M Referent  group)/Poolcd  SD,  (3) 

where  the  “referent  group”  is  males  for  gender  comparisons  and  whites  for  racial/ethnic 
comparisons. 

Gender  Differences 

CBEF  scale  means,  SDs,  and  Cohen’s  d  effect  sizes  for  gender  comparisons  appear  in 
Table  4.3.  As  shown,  effect  sizes  for  gender  on  the  RBI  and  PFC  scales  ranged  from  -.92  to 
.41  {Mean  d  =  -.09).  The  largest  effect  sizes  were  seen  for  (a)  Fitness  Motivation,  Stress 
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Tolerance,  Army  Identification  (favoring  males),  and  (b)  Achievement  and  Educational 
Identification  (favoring  females),  though  only  Fitness  Motivation  would  be  deemed  a  large 
effect  based  on  Cohen’s  (1988)  standards.  Near  zero  gender  differences  were  found  for  Peer 
Leadership,  Cognitive  Flexibility,  Impulsiveness,  and  the  Response  Distortion  scale.  Males 
tended  to  score  somewhat  better  than  females  on  the  PFC  scales  and  PFC,  but  these 
differences  were  small.  The  gender  differences  observed  in  this  sample  are  fairly  comparable 
to  past  research  involving  the  RBI.  For  example,  Kilcullen,  Putka,  &  McCloy  (2007)  found 
that  first-term  male  Soldiers  tended  to  have  higher  RBI  Fitness  Motivation,  Stress  Tolerance, 
and  Army  Identification  scores  relative  to  their  female  counterparts,  and  female  Soldiers 
tended  to  have  higher  Achievement  scores  relative  to  their  male  counterparts.  Thus,  the 
findings  here  regarding  gender  differences  tended  to  be  relatively  small,  or  at  least  fairly 
consistent  with  past  research;  the  only  sizable  difference  found  was  for  RBI  Fitness 
Motivation. 

Race/Ethnicity  Differences 

CBEF  scale  means,  SDs,  and  Cohen’s  d  effect  sizes  for  race/ethnicity  comparisons 
appear  in  Table  4.4.  As  shown,  effect  sizes  for  race/ethnicity  on  the  RBI  and  PFC  scales 
ranged  from  (a)  -.58  to  .32  for  Black- White  comparisons  ( Mean  d  =  -.05),  (b)  -.32  to  .34  for 
Hispanic-White  comparisons  ( Mean  d  =  -.07),  and  (c)  -.40  to  .51  for  Asian- White 
comparisons  (Mean  d  =  -.06).  With  regard  to  the  RBI,  moderate  effect  sizes  differences  were 
found  between  Black  and  White  Cadets  on  Army  Identification,  Stress  Tolerance,  and  Fitness 
Motivation — with  White  Cadets  tending  to  score  moderately  higher  (d  =  -.49  to  -.58).  In 
contrast.  Black,  Hispanic,  and  Asian  Cadets  tended  to  have  slightly  to  moderately  higher 
scores  on  RBI  Cultural  Tolerance  (d  =  .20  to  .51).  With  the  exception  of  the  differences 
noted  above,  only  small  differences  between  Cadets  of  different  race/ethnicities  were 
observed  on  the  RBI  scales. 

Like  the  gender  differences  described  above,  the  race/ethnicity  differences  bear 
resemblance  to  past  research  involving  the  RBI.  For  example,  Knapp  &  Tremble  (2007) 
found  that  White  first-term  Soldiers  tended  to  have  higher  RBI  Army  Identification  scores 
relative  to  Black  Soldiers,  but  unlike  the  current  sample,  White  and  Black  Soldiers  tended  to 
score  similarly  on  RBI  Stress  Tolerance  and  Fitness  Motivation.  Additionally,  similar  to  the 
present  sample  findings,  Kilcullen  et  al.  (2007)  found  that  Black  and  Hispanic  Soldiers 
tended  to  have  higher  scores  on  RBI  Cultural  Tolerance  relative  to  White  Soldiers.  With 
regard  to  WPS,  White  Cadets  were  found  to  have  moderately  higher  scores  than  both 
Hispanics  and  Blacks  (d  =  .40  and  -.57,  respectively),  and  slightly  higher  scores  than  Asians 
(d  =  -29).  Given  that  the  WPS  has  a  large  cognitive  ability  component  to  it  (e.g.,  College 
Board  scores),  and  summaries  of  past  research  have  demonstrated  Black-White  differences  of 
about  .83  standard  deviations  on  measures  of  cognitive  ability  (with  Whites  tending  to  score 
higher  than  Blacks;  Schmitt,  Clause,  &  Pulakos,  1996),  these  findings  are  well  within  what 
would  be  expected  of  past  research.  In  sum,  no  large  differences  between  races  on  either  the 
CBEF  or  WPS  were  found;  and  when  moderate  differences  were  found,  they  tended  to  be  in 
line  with  previous  research. 
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Table  4.3.  RBI,  PFC,  and  WPS  Means,  SDs,  and  Effect  Sizes  by  Gender 


Predictor 

N 

Male 

M 

SD 

N 

Female 

M 

SD 

F-M  d 

RBI:  Peer  Leadership  (PLEAD) 

808 

3.79 

.53 

226 

3.84 

.56 

.09 

RBI:  Cognitive  Flexibility  (CF) 

808 

3.65 

.61 

226 

3.61 

.63 

-.06 

RBI:  Achievement  Orientation  (ACH) 

808 

3.86 

.47 

226 

4.04 

.48 

.38 

RBI:  Fitness  Motivation  (FM) 

808 

3.95 

.58 

226 

3.44 

.52 

-.92 

RBI:  Interpersonal  Skills  -  Diplomacy  (ISDIP) 

808 

3.84 

.68 

226 

4.04 

.68 

.29 

RBI:  Stress  Tolerance  (ST) 

808 

3.14 

.44 

226 

2.88 

.49 

-.56 

RBI:  Hostility  to  Authority  (HOST) 

808 

1.95 

.46 

226 

1.84 

.40 

-.24 

RBI:  Self-Efficacy  (SE) 

808 

4.23 

.43 

226 

4.15 

.45 

-.19 

RBI:  Cultural  Tolerance  (CT) 

808 

3.88 

.62 

226 

4.06 

.60 

.29 

RBI:  Army  Identification  (AI) 

808 

4.05 

.61 

226 

3.83 

.67 

-.37 

RBI:  Impulsiveness  (IMP) 

808 

3.08 

.49 

226 

3.05 

.53 

-.07 

RBI:  Educational  Identification  (EDID) 

807 

4.38 

.49 

226 

4.58 

.43 

.41 

RBI:  Response  Distortion  (RD) 

808 

.07 

.11 

226 

.08 

.13 

.07 

PFC:  Persistence  (PERSIST) 

804 

40.17 

5.67 

226 

38.31 

5.18 

-.33 

PFC:  Group  Orientation  (GRORIENT) 

804 

40.79 

7.03 

226 

40.19 

6.76 

-.09 

Whole  Person  Score 

808 

238.93 

40.11 

226 

232.73 

40.36 

-.15 

Note.  F-M  d  =  (MPema —  MMaie)/Pooled  Female-Male  SD. 
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Table  4.4.  RBI,  PFC,  and  WPS  Means,  SDs,  and  Effect  Sizes  by  Race 


Predictor 

White 

Black 

Hispanic 

Asian 

d 

M 

SD 

M 

SD 

M 

SD 

M 

SD 

B-W 

H-W 

A-W 

RBI:  Peer  Leadership  (PLEAD) 

3.82 

.52 

3.83 

.58 

3.71 

.56 

3.71 

.61 

.02 

-.22 

-.21 

RBI:  Cognitive  Flexibility  (CF) 

3.64 

.62 

3.74 

.67 

3.54 

.67 

3.73 

.57 

.16 

-.16 

.14 

RBI:  Achievement  Orientation  (ACH) 

3.90 

.47 

3.96 

.49 

3.82 

.44 

3.89 

.51 

.11 

-.17 

-.04 

RBI:  Fitness  Motivation  (FM) 

3.86 

.60 

3.56 

.68 

3.77 

.68 

3.69 

.59 

-.50 

-.16 

-.28 

RBI:  Interpersonal  Skills  -  Diplomacy  (ISDIP) 

3.90 

.69 

3.91 

.74 

3.85 

.62 

3.85 

.65 

.02 

-.06 

-.06 

RBI:  Stress  Tolerance  (ST) 

3.11 

.46 

2.88 

.54 

3.02 

.48 

2.93 

.46 

-.49 

-.18 

-.40 

RBI:  Hostility  to  Authority  (HOST) 

1.92 

.45 

1.98 

.47 

2.00 

.50 

1.89 

.47 

.12 

.16 

-.07 

RBI:  Self-Efficacy  (SE) 

4.23 

.43 

4.15 

.48 

4.09 

.50 

4.14 

.48 

-.18 

-.32 

-.20 

RBI:  Cultural  Tolerance  (CT) 

3.89 

.63 

4.01 

.62 

4.10 

.57 

4.20 

.49 

.20 

.34 

.51 

RBI:  Army  Identification  (AI) 

4.03 

.62 

3.67 

.75 

4.01 

.64 

3.89 

.68 

-.58 

-.04 

-.22 

RBI:  Impulsiveness  (IMP) 

3.07 

.50 

3.23 

.54 

3.11 

.45 

3.10 

.55 

.32 

.08 

.07 

RBI:  Educational  Identification  (EDID) 

4.42 

.49 

4.48 

.42 

4.31 

.48 

4.37 

.49 

.12 

-.22 

-.10 

RBI:  Response  Distortion  (RD) 

.07 

.12 

.08 

.12 

.09 

.12 

.09 

.14 

.06 

.13 

.11 

PFC:  Persistence  (PERSIST) 

39.96 

5.51 

38.62 

5.91 

38.98 

6.34 

38.96 

5.20 

-.24 

-.18 

-.18 

PFC:  Group  Orientation  (GRORIENT) 

40.65 

6.78 

41.85 

7.85 

40.27 

7.43 

40.64 

7.32 

.18 

-.06 

.00 

Whole  Person  Score 

240.97 

38.31 

219.04 

37.97 

225.56 

41.34 

229.69 

39.37 

-.57 

-.40 

-.29 

Note.  B-W  d  =  (MBiack  —  Mwhite)/Pooled  Black-White  SI).  H-W  d  =  (MHispanic  —  Mwhite)/Pooled  Hispanic -White  SO.  A-W  d  =  (MAsim  —  Mwhite)/Pooled  Asian- 
White  SD.  Ns  were  as  follows:  879  to  911  White,  53  to  55  Black,  64  to  68  Hispanic,  and  80  to  84  Asian. 


22 


Basic  Properties  of  Key  Criterion  Measures 


In  the  sections  that  follow,  we  describe  the  three  criterion  variables  used  to  evaluate 
the  predictive  potential  of  the  CBEF.  The  criteria  include  a  Cadet’s  enrollment  status  (i.e., 
disenrolled  vs.  enrolled  as  of  October  2007)  and  two  self-reported  likelihood  variables  that 
were  included  as  part  of  the  CBEF  administered  in  the  Spring  of  2007  (see  Chapter  2). 

Cadet  Enrollment  Status 

As  described  in  Chapter  2,  the  Student  Management  Database  contained  variables 
indicating  whether  Cadets  disenrolled  from  the  ROTC  program  between  their  completion  of 
the  CBEF  in  the  Spring  of  2007,  and  October  2007.  Among  those  Cadets  who  completed  the 
CBEF  in  the  Spring  of  2007  and  had  valid  records  in  the  Student  Management  Database,  we 
found  that  925  (89.7%)  were  still  enrolled  as  of  October  2007,  and  106  (10.3%)  had 
disenrolled  as  of  October  2007. 5 

Based  on  data  provided  by  Cadet  Command,  we  were  able  to  take  a  finer  grained  look 
at  those  Cadets  who  disenrolled.  Table  4.5  presents  the  frequencies  for  the  disenrollment 
codes  for  these  Cadets.6  Over  half  of  this  sample  was  noted  as  disenrolling  by  their  “own 
request.”  Such  a  reason  sheds  little  light  on  why  Cadets  requested  to  disenroll  or,  pertinent  to 
this  effort — can  characteristics  assessed  by  the  CBEF  help  identify  what  types  of  Cadets  are 
most  likely  to  disenroll?  In  Chapter  5,  we  summarize  analyses  designed  to  determine  whether 
Cadets’  scores  on  the  RBI  and  PFC  components  of  the  CBEF  were  able  to  predict  which 
Cadets’  disenrolled  as  of  October  2007. 


Table  4.5.  Frequency  of  Reasons  for  Disenrollment 


Reason  Code 

N 

% 

D01:  Withdrawal  from  school 

15 

14.15 

D03:  Academic  Failure 

5 

4.72 

Dll:  Transfer  to  another  ROTC  program  (e.g..  Air  Force) 

1 

0.94 

D12:  Failure  to  maintain  enrollment  requirements 

2 

1.89 

D13:  Own  request 

63 

59.43 

D15:  Transfer  to  another  ROTC  Battalion 

6 

5.66 

D23:  Volunteer  Release 

2 

1.89 

D37:  Withdrew  from  ROTC 

11 

10.38 

D39:  Academic  Dismissal/Suspension 

1 

0.94 

Total 

106 

100.00 

5  In  Chapter  3  we  noted  there  were  1,068  Cadets  in  the  validation  database.  The  numbers  cited  here  add  up  to 
1,031.  This  difference  is  explained  by  the  fact  that  of  the  1,068  Cadets  in  the  validation  database  (a)  35  had  no 
records  in  the  Student  Management  Database,  and  (b)  2  Cadets  had  disenrollment  dates  suggesting  they 
disenrolled  prior  to  completing  the  CBEF  (logically  impossible).  As  such,  the  aforementioned  37  Cadets  were 
excluded  when  calculating  the  disenrollment  criterion  variable. 

6  The  disenrollment  codes  presented  here  should  be  interpreted  very  carefully  because  it  has  been  noted  that 
disenrollment  reasons  are  not  reliably  and  accurately  recorded  for  all  Cadets  (e.g..  Baker,  2006).  As  such,  we  do 
not  use  these  codes  for  any  subsequent  analyses.  We  present  them  here  for  informational  purposes  only. 
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Self-Rated  Likelihood  Items  from  the  CBEF 


Although  the  disenrollment  criterion  variable  was  of  primary  interest  in  the  current 
effort,  there  are  other  criteria  that  Cadet  Command  may  wish  to  consider  when  awarding 
scholarships — specifically,  the  likelihood  that  a  prospective  Cadet  will  eventually  become  an 
Army  Officer  and  make  the  Army  a  career.  Unfortunately,  establishing  a  direct  relationship 
between  RBI  and  PFC  scales  from  the  CBEF  and  these  variables  is  not  possible  based  on  the 
current  data,  as  it  will  take  several  years  to  know  which  of  today’s  Cadets  will  eventually 
become  Officers,  and  in  the  case  of  making  the  Army  a  career,  may  take  several  decades.  In 
light  of  the  limitations  of  the  current  data,  we  included  two  questions  on  the  CBEF  that  were 
designed  to  serve  as  proxies  for  these  long-term  events,  self-rated  likelihood  of  (a)  becoming 
an  Army  Officer  (i.e.,  completing  ROTC)  and  (b)  making  the  Army  a  career. 

Cadets  were  asked  to  rate  the  likelihood  of  each  event  described  above  on  a  five-point 
Likert-type  scale  (l=Very  Unlikely  to  5=Very  Likely).  Analyses  of  Cadets’  responses 
revealed  that  they  were  highly  negatively  skewed  (i.e.,  most  Cadets  chose  a  response  of 
Likely  and  Very  Likely).  Therefore,  the  response  scales  were  dichotomized  for  purposes  of 
subsequent  validation  analyses.  The  specific  dichotomization  differed  for  the  two  scales 
because  they  had  different  distributions  of  responses.  Lor  example,  in  the  case  of  the  self- 
rated  likelihood  of  becoming  an  Army  Officer  variable,  we  believed  it  made  more  sense  to 
group  Cadets  who  responded  using  a  rating  of  1,  2,  or  3  into  the  same  category  because  the 
number  of  Cadets  who  had  any  one  of  these  responses  was  very  low.  In  contrast,  the  “break” 
in  the  distribution  for  the  likelihood  of  making  the  Army  Career  variable  appeared  to  come 
between  ratings  of  2  and  3,  as  a  relatively  small  number  of  Cadets  responded  using  a  rating 
of  1  or  2  on  this  variable,  and  a  more  substantial  number  of  Cadets  responded  using  ratings  of 
3,  4,  or  5.  As  such,  we  recoded  the  self-rated  likelihood  of  becoming  an  Army  Officer 
variable  as:  (a)  “More  Likely”  for  Cadets  who  responded  using  a  5  or  4  on  the  rating  scale 
(i.e.,  Very  Likely  or  Likely)  or  (b)  “Less  Likely”  for  Cadets  who  responded  using  a  3,  2,  or  1 
on  the  rating  scale  (i.e.,  Somewhat  Likely,  Unlikely,  or  Very  Unlikely).  We  recoded  the  self- 
rated  likelihood  of  making  the  Army  a  career  variable  as:  (a)  “Likely”  for  Cadets  who 
responded  using  a  5,  4,  or  3  on  the  rating  scale  (i.e.,  Very  Likely,  Likely  or  Somewhat 
Likely)  or  (b)  “Unlikely”  for  Cadets  who  responded  using  a  2  or  1  on  the  rating  scale  (i.e., 
Unlikely  or  Very  Unlikely).  Table  4.6  summarizes  Cadet’s  standing  on  the  dichotomized 
self-rated  likelihood  criterion  variables. 

Table  4.6.  Summary  Statistics  for  Self-Rated  Likelihood  Criterion  Variables 


N 

% 

Self-Rated  Likelihood  of  Becoming  an  Army  Officer 

Less  Likely  59 

5.78 

More  Likely  96 1 

94.22 

Self-Rated  Likelihood  of  Making  the  Army  a  Career 

Unlikely  101 

9.90 

Likely  919 

90.10 
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Although  only  a  small  portion  of  Cadets  reported  being  less  likely  to  become  an 
Army  Officer  (5.78%)  or  unlikely  to  make  the  Army  a  career  (9.90%),  it  is  noteworthy  that 
even  this  many  Cadets  admitted  their  intentions  so  early  in  their  ROTC  experience.  In 
Chapter  5,  we  summarize  analyses  designed  to  determine  whether  Cadets’  scores  on  the  RBI 
and  PFC  components  of  the  CBEF  were  able  to  predict  Cadets’  self-rated  likelihood  of  these 
events. 


As  discussed  previously,  the  self-reported  likelihood  variables  offer  a  possible 
alternative  to  actual  measures  of  Cadets’  behavior  (e.g.,  becoming  an  Army  Officer)  when 
practical  constraints  cause  the  latter  data  to  be  unavailable.  For  example,  we  will  not  know  if 
Cadets  who  completed  the  CBEF  in  the  Spring  of  2007  will  actually  become  Army  Officers 
for  at  least  three  more  years.  Thus,  these  likelihood  variables  can  provide  proxies  for  actual 
later  behavior.  To  help  assess  the  quality  of  the  proxies  these  variables  provide,  we  examined 
their  relationship  to  Cadets’  enrollment  status  as  of  October  2007.  As  seen  in  Table  4.7, 
72.41%  of  Cadets  who  indicated  that  they  were  less  likely  to  become  an  Army  Officer, 
subsequently  disenrolled.  In  contrast,  only  6.36%  of  Cadets  who  indicated  they  were  more 
likely  to  become  an  Army  Officer,  subsequently  disenrolled,  This  pattern  of  findings 
suggests  a  strong  relationship  between  self-reported  likelihood  to  become  an  Army  Officer 
and  actual  disenrollment  behavior — even  though  in  the  present  sample,  such  behavior 
potentially  occurred  several  months  later. 

Table  4. 7.  Enrollment  Status  by  Self-Reported  Likelihood  of  Becoming  an  Army 
Officer 

Likelihood  of  Becoming  an 


Status 

Army  Officer 

Total 

N 

More  Likely 

Less  Likely 

N 

% 

N 

% 

Enrolled 

868 

93.63 

16 

27.59 

Disenrolled 

49 

6.36 

42 

72.41 

Total 

927 

- 

58 

- 

985 

Table  4.8  presents  parallel  results  for  the  likelihood  of  making  the  Army  a  career 
criterion  variable.  The  results  again  indicate  a  strong  relationship  between  self-rated 
likelihood  and  subsequent  behavior.  For  example,  only  7.55%  of  Cadets  who  said  they  were 
likely  to  make  the  Army  a  career  subsequently  disenrolled  compared  to  35.05%  of  those  who 
said  they  were  unlikely  to  make  the  Army  a  career.  Although  this  latter  number  is  lower  than 
the  72.41%  disenrollment  rate  among  Cadets  who  indicated  they  were  less  likely  to  become 
an  Army  Officer,  it  is  important  to  note  the  difference  in  the  substantive  content  inferred  by 
the  two  questions.  Specifically,  all  else  being  equal,  we  would  expect  there  to  be  a  stronger 
relation  between  disenrollment  and  self-rated  likelihood  of  becoming  an  Army  Officer  (i.e., 
likelihood  of  not  disenrolling)  relative  to  more  general  likelihood  of  making  the  Army  a 
career — which  may  be  determined  by  other  factors  that  do  not  manifest  themselves  until  after 
a  Cadet  graduates  (e.g.,  quality  of  available  civilian  job  opportunities). 
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Table  4.8.  Enrollment  Status  by  Self-Reported  Likelihood  of  Making  the  Army  a 
Career 


Status 

Likelihood  of  Making  the 

Army  a  Career 

Total 

N 

Likely 

Unlikely 

N 

% 

N 

% 

Enrolled 

821 

92.45 

63 

64.94 

884 

Disenrolled 

67 

7.55 

34 

35.05 

101 

Total 

888 

- 

97 

- 

985 
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Chapter  5.  Criterion-Related  Validation  Results 

Shonna  Waters  and  Dan  J.  Putka 
HumRRO 

This  chapter  presents  criterion-related  validity  evidence  with  the  intention  of 
evaluating  the  potential  of  the  CBEF’s  RBI  and  PFC  scales  to  predict  disenrollment  of  four- 
year  scholarship  recipients,  as  well  as  their  self-rated  likelihood  of  becoming  an  Army 
Officer  and  making  the  Army  a  career.  Among  the  topics  discussed  here  are  correlations 
between  the  CBEF  scales  and  criteria,  logistic  regression  results,  and  a  discussion  of  the 
potential  impact  that  including  the  CBEF  may  have  on  Cadet  disenrollment  if  it  were 
included  as  part  of  the  scholarship  awarding  process.  We  also  report  on  relations  between 
Whole  Person  Scores  (WPS)  and  the  criteria  to  provide  a  point  of  comparison  for  evaluating 
relations  between  RBI  and  PFC  scales  and  the  criteria. 

Bivariate  Relations  between  Predictor  Scales  and  Criterion  Variables 

Table  5.1  presents  point-bi serial  correlations  between  CBEF  predictor  scales  and 
criterion  variables.  Note  that  interpretation  of  the  magnitude  of  a  point-biserial  correlation 
must  take  into  account  the  base  rate  of  the  dichotomous  variable  involved  in  the  correlation 
(e.g.,  the  percentage  of  Cadets  who  disenroll).  Deviations  of  the  base  rate  from  50%  depress 
the  maximum  size  of  the  correlation  that  be  obtained.  The  10.3%  base  rate  of  disenrollment 
in  the  current  dataset  limited  the  maximum  observable  point-bi  serial  correlation  to  .59  for 
disenrollment.  The  maximum  correlations  for  the  dichotomized  likelihood  variables  were  .50 
and  .59  for,  respectively,  becoming  an  Army  Officer  and  making  the  Army  a  career.  In  light 
of  the  sensitivity  of  point-biserial  correlations  to  base  rate,  Table  5.1  also  presents  Cohen’s  cl 
effect  sizes  for  each  predictor  scale-criterion  combination  (Cohen,  1988).  Unlike  the  point- 
biserial  correlations,  the  magnitude  of  Cohen’s  d  is  not  affected  by  the  base  rate  issues  and 
simply  reflects  the  standardized  mean  difference  between  predictor  scale  scores  for  each 
group  defined  by  a  given  criterion  variable  (e.g.,  disenrolled  Cadets  vs.  enrolled  Cadets, 
Cadets  who  indicated  they  were  likely  to  make  the  Army  a  career  vs.  Cadets  who  indicated 
they  were  unlikely  to  make  the  Army  a  career).  Cohen’s  cl  was  calculated  as  follows: 

Disenrollment  criterion: 

d  =  (MDisenroiied  Cadets  Tfcmollecl  Cadets)/Poolcd  SD  (1) 

Fikelihood  of  becoming  an  Army  Officer  criterion: 

cl  =  ((If “Less  likely”  Cadets  df“More  likely”  Cadets)/Pooled  SD  (2) 

Fikelihood  of  making  the  Army  a  career  criterion: 

d  —  ((If “Unlikely”  Cadets  df“Likely”  Cadets)/Pooled  SD  (3) 


Two  types  of  Cohen’s  d  effect  sizes  are  provided  for  each  criterion  in  Table  5.1.  The  first 
type,  simply  labeled  “d”,  reflects  raw  effect  sizes  unadjusted  for  range  restriction  stemming 
from  selection  of  Cadets  on  the  WPS.  The  second  type,  labeled  “d",  reflects  corrected  effect 
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Table  5.1.  Bivariate  Relations  between  Predictors  and  Criteria 


Predictor 

Disenrollment 

Likelihood  to  Become 
an  Army  Officer 

Likelihood  to  Make 
the  Army  Career 

r0  b 

d 

dc 

r  Db 

d 

dc 

r  ob 

d 

dc 

Whole  Person  Score  (WPS) 

-.04 

-.12 

-.32 

-.01 

.05 

.13 

-.11 

.38 

1.01 

RBI:  Peer  Leadership  (PLEAD) 

-.02 

-.08 

-.14 

.09 

-.37 

-.33 

.08 

-.26 

-.05 

RBI:  Cognitive  Flexibility  (CF) 

-.01 

-.04 

-.12 

.05 

-.19 

-.15 

.02 

-.07 

.17 

RBI:  Achievement  Orientation  (ACH) 

-.09 

-.30 

-.35 

.15 

-.63 

-.59 

.12 

-.39 

-.19 

RBI:  Fitness  Motivation  (FM) 

-.10 

-.32 

-.35 

.10 

-.41 

-.40 

.09 

-.29 

-.19 

RBI:  Interpersonal  Skills  -  Diplomacy  (ISDIP) 

.02 

.06 

.06 

.06 

-.25 

-.25 

.03 

-.11 

-.11 

RBI:  Stress  Tolerance  (ST) 

-.04 

-.13 

-.18 

.11 

-.47 

-.44 

.07 

-.25 

-.08 

RBI:  Hostility  to  Authority  (HOST) 

.07 

.24 

.27 

-.09 

.38 

.36 

-.09 

.30 

.18 

RBI:  Self-Efficacy  (SE) 

-.09 

-.30 

-.34 

.13 

-.56 

-.54 

.16 

-.54 

-.38 

RBI:  Cultural  Tolerance  (CT) 

.01 

.02 

-.02 

.01 

-.03 

-.01 

.07 

-.24 

-.12 

RBI:  Army  Identification  (AI) 

-.19 

-.65 

-.61 

.32 

-1.44 

-1.45 

.40 

-1.45 

-1.49 

RBI:  Impulsiveness  (IMP) 

.08 

.28 

.33 

-.11 

.48 

.45 

-.05 

.17 

.01 

RBI:  Educational  Identification  (EDID) 

-.11 

-.38 

-.46 

.13 

-.58 

-.49 

.04 

-.14 

.21 

RBI:  Response  Distortion  (RD) 

.05 

.15 

.22 

.01 

-.03 

-.06 

.07 

-.23 

-.43 

PFC:  Persistence  (PERSIST) 

-.08 

-.25 

-.27 

.11 

-.49 

-.48 

.12 

-.41 

-.33 

PFC:  Group  Orientation  (GRORIENT) 

-.03 

-.09 

-.02 

.08 

-.32 

-.34 

.06 

-.22 

-.40 

Note:  N  =  995-1020.  rpb  =  Point-biserial  correlation,  d  =  Raw  Cohen’s  d  statistic,  d,  =  Corrected  Cohen’s  d  statistic.  Boldface  text  indicates  correlation  was 
significant  (p  <.05,  one-tailed). 
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sizes  that  have  been  adjusted  for  range  restriction  stemming  from  selection  of  Cadets  on  the 
WPS.  Recall  from  Chapter  2,  that  there  was  a  substantial  amount  of  range  restriction  in  WPS 
among  Cadets  in  the  current  sample.  Specifically,  for  example,  the  standard  deviation  of 
WPS  scores  of  freshmen  completing  the  CBEF  in  the  current  sample  was  40.22,  whereas  the 
standard  deviation  of  WPS  among  all  SY06-07  applicants  was  107.24.  Thus,  in  an  attempt  to 
provide  a  fairer  comparison  of  the  magnitude  of  relations  between  RBI/PFC  scales  and  the 
criteria,  relative  to  the  WPS  and  the  criteria,  effects  sizes  were  corrected  for  range  restriction 

7 

using  formulae  provided  by  Bobko,  Roth,  &  Bobko  (2001). 

Based  on  Table  5.1  the  strongest  predictors  of  disenrollment  appeared  to  be  RBI 
Army  Identification  (dc  =  -.61)  and  Educational  Identification  (d,  =  -.46),  indicating  that 
Cadets  who  disenrolled  scored  over  Vi  of  a  standard  deviation  lower  on  these  scales  than 
those  who  continued  enrollment.  The  other  RBI  and  PFC  scales  with  statistically  significant 
but  relative  small  relations  with  disenrollment  were  RBI  Achievement,  Fitness  Motivation, 
Hostility  to  Authority,  Self-Efficacy,  Impulsiveness,  and  PFC  Persistence  (dc‘s  ranging  from 
.27.  to  .35  in  magnitude).  We  should  note  that  although  unadjusted  WPS  scores  were  not 
significantly  related  to  disenrollment,  WPS  scores  adjusted  for  range  restriction  exhibited  an 
effect  size  of  -.32  (comparable  to  the  aforementioned  RBI  and  PFC  scales).  Such  an  effect 
size  translates  into  a  WPS  standard  deviation  for  disenrolling  Cadets  that  was  about  1/3  lower 
than  the  standard  deviation  for  Cadets  remaining  in  the  ROTC  program. 

With  regard  to  the  two  likelihood  criteria,  several  notable  findings  emerged.  First, 
there  was  an  extremely  large  relation  observed  between  the  RBI  Army  Identification  scale 
and  both  likelihood  criteria.  For  example,  Cadets  who  indicated  they  were  likely  to  make  the 
Army  a  career  had  RBI  Army  Identification  scores  that  were  1.49  standard  deviations  higher 
than  scores  for  Cadets  who  indicated  they  were  unlikely  to  make  the  Army  a  career. 

Moderate  relations  were  found  between  several  RBI/PFC  scales  and  the  likelihood  to  become 
an  Army  Officer  criterion;  namely,  RBI  Achievement,  Fitness  Motivation,  Stress  Tolerance, 
Self-Efficacy,  Impulsiveness,  Educational  Identification,  and  PFC  Persistence  all  exhibited 
statistically  significant  levels  of  criterion-related  validity  with  relations  ranging  from  .40  to 
.59  in  magnitude  (based  on  corrected  d’s).  In  general,  the  pattern  of  the  most  predictive  RBI- 
PFC  scales  for  the  likelihood  to  become  an  Army  Officer  resembled  that  for  disenrollment. 
Such  findings  help  reinforce  the  idea  that  this  self-rated  likelihood  variable  may  serve  as  a 
useful  proxy  when  evaluating  the  relative  merit  of  potential  predictors  of  disenrollment. 

For  the  likelihood  to  make  the  Army  a  career  criterion,  many  statistically  significant 
relations  were  observed,  but  the  effects  (with  the  exception  of  RBI  Army  Identification  noted 
above)  were  far  less  strong.  Some  different  scales  also  came  up  as  significant.  For  example, 
after  RBI  Army  Identification,  the  two  strongest  RBI-PFC  based  predictors  of  a  Cadet’s 
likelihood  of  making  the  Army  a  career  were  the  RBI  Response  Distortion  and  the  PFC 
Group  Orientation  scale — in  both  of  these  cases  Cadets  who  indicated  they  were  likely  to 
make  the  Army  a  career  tended  to  have  moderately  higher  scores  on  these  scales  relative  to 
Cadets  who  indicated  they  were  unlikely  to  make  the  Army  a  career.  A  key  finding  to  note 


7  Note,  when  correcting  effect  sizes  for  the  WPS,  we  used  Bobko  et  al.’s  (2001)  formula  for  direct  range 
restriction,  and  when  correcting  effects  sizes  for  RBI  and  PFC  scales,  we  used  Bobko  et  al.’s  (2001)  formula  for 
indirect  range  restriction. 
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here  concerns  the  effect  that  correcting  for  range  restriction  on  the  WPS  had  for  the  RBI/PFC 
-  likelihood  of  making  the  Army  career  relationships.  Specifically,  implementing  the 
correction  tended  to  reduce  the  magnitude  of  the  relationship  observed  for  several  RBI/PFC 
scales.  This  seemingly  unintuitive  finding  can  be  explained  by  the  observation  that  the  WPS 
had  a  strong  negative  relation  with  the  likelihood  of  making  an  Army  a  career.  Thus,  after 
having  accounted  for  range  restriction  on  the  WPS,  Cadets  likely  to  make  the  Army  a  career 
had  WPS  scores  which  were  about  one  standard  deviation  lower  than  those  for  Cadets  who 
were  unlikely  to  make  the  Army  a  career.  Such  findings  stand  in  contrast  to  the  more  positive 
findings  regarding  the  relation  between  WPS  and  disenrollment  presented  above. 

Multivariate  Relations  between  Predictor  Scales  and  Criterion  Variables 

In  addition  to  examining  bivariate  relations  between  each  predictor  scale  and 
criterion,  we  also  conducted  a  series  of  logistic  regression  analyses  to  examine  multivariate 
relations  between  predictors  and  criteria.  Specifically,  such  analyses  were  performed  to 
examine  the  extent  to  which  (a)  composites  of  RBI  and  PFC  scales  could  predict  each 
criterion,  and  (b)  RBI  and  PFC  scales  could  increment  the  validity  of  the  WPS  for  predicting 
each  criterion. 

Five  logistic  regression  models  were  fitted  for  each  criterion.  In  all  models,  the 
predictors  were  standardized  prior  to  being  entered  into  the  logistic  regression. 

•  The  first  model  included  the  WPS  as  the  only  predictor  of  the  given  criterion. 
The  purpose  of  this  model  was  to  establish  a  baseline  for  comparing  models 
that  included  RBI  and  PFC  scales. 

•  The  second  model  included  only  the  RBI  and  PFC  scales  as  predictors  of  the 
given  criterion.  The  purpose  of  this  model  was  to  (a)  assess  the  level  of 
criterion-related  validity  achievable  with  a  regression-weighted  composite  of 
RBI  and  PFC  scales  and  (b)  assess  the  significance  of  each  RBI  and  PFC  scale 

o 

when  considered  in  a  model  that  included  the  other  RBI  and  PFC  scales. 

•  The  third  model  included  the  WPS,  RBI,  and  PFC  scales  as  predictors  of  the 
given  criterion.  The  purpose  of  this  model  was  to  (a)  assess  the  level  of 
criterion-related  achievable  with  a  regression-weighted  composite  of  WPS,  RBI 
and  PFC  scales  and  (b)  assess  the  significance  of  each  RBI  and  PFC  scales 
relative  to  the  WPS. 

•  The  fourth  model  included  a  unit- weighted  composite  of  RBI  and  PFC  scales  as 
the  sole  predictor  of  the  given  criterion. 

•  Last,  the  fifth  model  included  the  WPS  and  the  aforementioned  unit-weighted 
composite  of  RBI  and  PFC  scales  as  the  sole  predictor  of  the  given  criterion. 


8  Note,  the  “regression-weighted  composites”  alluded  to  here  and  in  the  next  bullet  simply  reflect  the  predicted 
values  resulting  from  fitting  the  models  described  in  these  bullets. 
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The  fourth  and  fifth  models  differed  from  the  second  and  third  models  in  two  notable 
ways.  First,  they  did  not  include  each  RBI  and  PFC  scale  as  a  separate  predictor,  but  rather 
included  a  single  unit- weighted  composite  of  each  set  of  scales.  One  potential  criticism  of 
drawing  conclusions  based  solely  on  the  performance  of  regression- weighted  composites  of 
RBI  and  PFC  scales  (i.e.,  predicted  values  resulting  from  models  where  each  RBI  and  PFC 
scale  was  entered  as  a  separate  predictor)  is  that  combining  scales  in  such  a  manner 
capitalizes  on  chance.  Specifically,  when  one  regresses  a  criterion  on  a  set  of  predictors,  the 
regression  coefficients  (weights)  associated  with  those  predictors  are  “optimized”  based  on 
the  sample  on  which  the  model  is  fitted.  All  else  being  equal,  if  those  weights  were  applied  to 
the  predictor  scores  (e.g.,  the  RBI-PFC  scales)  for  another  sample,  the  criterion-related 
validity  of  the  resulting  composite  would  tend  to  be  lower  than  the  criterion-related  validity 
of  the  composite  in  which  the  weights  were  initially  estimated.  As  such,  the  first  purpose  of 
examining  models  four  and  five  was  to  assess  the  performance  of  RBI/PFC  scales  based  on  a 
composite  that  was  not  optimized  based  on  the  sample  dataset. 

A  second  difference  between  the  fourth  and  fifth  models  relative  to  the  second  and 
third  models  is  that  rather  than  including  all  RBI  and  PFC  scales,  the  fourth  and  fifth  models 
included  only  those  scales  that  were  to  be  carried  forward  in  a  refined  version  of  the  CBEF 
for  uses  in  later  efforts  on  Cadet  selection  for  retention  (see  Chapter  6).  The  RBI  and  PFC 
scales  included  in  this  reduced  composite  were:  RBI  Self-Efficacy,  Stress  Tolerance, 
Achievement  Orientation,  Army  Identification,  Fitness  Motivation,  Educational 
Identification,  Hostility  to  Authority,  and  PFC  Group  Orientation.  Thus,  examination  of 
models  four  and  five  allowed  assessment  of  a  composite  formed  from  the  RBEPFC  scales 
that  would  be  available  after  refinement  of  the  CBEF. 

We  summarize  results  of  logistic  regression  analyses  for  each  criterion  using  two 
types  of  result  tables.  The  first  table  for  a  criterion  summarizes  results  for  Models  1  through 
3,  and  the  second  table  for  the  criterion  summarizes  results  for  Models  4  and  5  (with  Model  1 
repeated  for  purposes  of  comparison).  Tables  present  odds  ratios  for  each  predictor  in  the 
models  of  interest.  Odds  ratios  above  one  indicate  that  an  increase  on  the  predictor  variables 
is  associated  with  an  increase  in  the  odds  that  a  Cadet  will  experience  the  event  of  interest 
(e.g.,  disenrollment,  likely  to  become  an  Officer,  likely  to  make  the  Army  a  career).  Odds 
ratios  less  than  one  indicate  that  an  increase  on  the  predictor  is  associated  with  a  decrease  in 
the  odds  that  a  Cadet  will  experience  the  event  of  interest.  For  example,  if  the  criterion  of 
interest  was  disenrollment,  an  odds  ratio  of  1.5  for  a  given  predictor  would  indicate  that  for 
every  one  unit  increase  on  that  predictor  (holding  values  for  other  predictors  in  the  model 
constant),  there  would  be  an  associated  50%  increase  in  the  odds  of  disenrollment — i.e.,  the 
odds  of  disenrollment  would  increase  by  1.5  times.  Conversely,  if  the  odds  ratio  was  0.50  for 
a  given  predictor,  it  would  indicate  that  for  every  one  unit  increase  on  that  predictor  (again, 
holding  values  on  other  predictors  in  the  model  constant),  the  odds  of  disenrollment  would  be 
halved.  In  addition  to  odds  ratios,  result  tables  also  provide  Cohen’s  d  statistic  for  predicted 
criterion  values  based  on  each  model.  These  values  reflect  the  standardized  mean  difference 
between  predicted  scores  for  Cadets  who  experience  the  event  of  interest  (e.g., 
disenrollment)  and  those  who  do  not.  Specifically,  Cohen’s  d  was  calculated  as  follows  for 
each  criterion: 
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Disenrollment  criterion: 

cl  —  ( M Predicted  Score  for  Disenrolled  Cadets  ■^Predicted  Score  for  Enrolled  Cadets)/Pooled  SD  (3) 

Likelihood  of  becoming  an  Army  Officer  criterion: 

d  =  (M  Predicted  Score  "More  likely”  Cadets  M  Predicted  Score  “Less  likely”  Cadets)/Pooled  SD  (4) 

Likelihood  of  making  the  Army  a  career  criterion: 

d  =  ( M  Predicted  Score  "Likely”  Cadets  4/ Predicted  Score  “Unlikely”  Cadets)/Pool*-d  SD  (5) 


Regression  Results  for  Predicting  Cadet  Disenrollment 

Results  presented  in  Table  5.2  reveal  a  number  of  notable  findings.  First,  note  that  the 
strongest  predictors  of  disenrollment  appear  to  be  RBI  Army  Identification  and  Educational 
Identification.  For  example,  for  every  one  standard  deviation  increase  in  RBI  Army 
Identification  scores,  there  was  a  69%  decrease  in  the  odds  that  a  Cadet  would  disenroll 
(holding  all  other  predictors,  including  the  WPS,  constant).9  For  every  one  standard 
deviation  increase  in  RBI  Educational  Identification  scores,  there  was  a  35%  decrease  in  the 
odds  that  a  Cadet  would  disenroll  (again,  holding  all  other  predictors  constant).  The  only 
other  RBI/PFC  scale  that  was  significantly  related  to  disenrollment  when  accounting  for  the 
effects  of  the  other  predictors  was  the  RBI  Response  Distortion  scale.  Specifically,  for  every 
one  standard  deviation  increase  in  RBI  Response  Distortion  scale  scores,  there  was  a  31% 
increase  in  the  odds  that  a  Cadet  would  disenroll  (again,  holding  all  other  predictors 
constant).  Though  the  Response  Distortion  scale  was  not  explicitly  designed  to  be  a 
“predictor”  of  disenrollment,  such  findings  may  suggest  that  high  scores  on  this  scale  are 
acting  as  a  proxy  for  deviance  or  a  propensity  to  misrepresent  oneself — which  in  turn,  may 
be  theoretically  related  to  increased  risk  for  disenrollment  from  ROTC  between  one’s 
freshman  and  sophomore  years. 

The  other  finding  of  note  in  Table  5.2  is  the  increase  in  criterion-related  validity  that 
occurred  when  the  RBI  and  PFC  scales  were  added  to  a  model  containing  the  WPS  only 
(e.g.,  Model  1  d=  .14  vs.  Model  3  cl  =  .99). 10  A  comparison  of  chi-squared  fit  statistics  for 
these  models  revealed  that  this  increment  in  validity  was  statistically  significant  (Ax'  [15]  = 
60.40,  p  <  .01,  one-tailed).  Further  examination  revealed  that  this  improvement  in  validity 
was  attributable  to  the  RBI  scales.  Specifically,  comparison  of  Model  3  to  a  model  with 
only  the  WPS  and  RBI  scales  as  predictors  revealed  that  addition  of  the  PFC  scales  to  the 
model  failed  to  significantly  increment  its  validity  relative  to  the  reduced  model  (Ax'  [2]  = 
0.46,  ns,  one-tailed). 


9  When  odds  ratios  are  less  than  one,  it  is  often  convenient  to  “invert”  them  (i.e.,  I/OR )  for  purposes  of 
interpretation.  Thus,  the  69%  value  we  report  here  for  RBI  Army  Identification  was  obtained  by  taking  the  OR 
associated  with  Army  Identification  in  Table  5.2  (OR  =  0.59)  and  inverting  it,  and  expressing  it  as  a  percentage 
change  relative  1.00  (i.e.,  1/.59  =  1.69,  which  equates  to  a  69%  change). 

10  Please  note,  the  incremental  validity  results  presented  here  and  in  subsequent  sections  do  not  account  for 
range  restriction  on  the  WPS.  Thus,  these  results  may  be  viewed  as  upper  bounds  for  the  potential  incremental 
validity  of  the  RBI/PFC  scales  over  the  WPS.  To  our  knowledge,  no  method  currently  exists  for  implementing 
multivariate  range  restriction  corrections  when  one’s  criterion  (e.g.,  disenrollment)  is  dichotomous  in  nature. 
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Table  5.2.  Logistic  Regression  Results  with  Disenrollment  as  the  Criterion 


Predictor 

Odds  Ratios 

Model  1 
(WPS  Only) 

Model  2 
(RBI+PFC) 

Model  3 

(WPS+RBI+PFC) 

Whole  Person  Score  (WPS) 

.88 

- 

.92 

RBI:  Peer  Leadership  (PLEAD) 

- 

1.14 

1.15 

RBI:  Cognitive  Flexibility  (CF) 

- 

1.01 

1.02 

RBI:  Achievement  Orientation  (ACH) 

_ 

.94 

.94 

RBI:  Fitness  Motivation  (FM) 

_ 

.86 

.86 

RBI:  Interpersonal  Skills  -  Diplomacy  (ISDIP) 

- 

1.22 

1.22 

RBI:  Stress  Tolerance  (ST) 

- 

1.02 

1.03 

RBI:  Hostility  to  Authority  (HOST) 

- 

1.08 

1.08 

RBI:  Self-Efficacy  (SE) 

- 

.84 

.84 

RBI:  Cultural  Tolerance  (CT) 

_ 

1.11 

1.11 

RBI:  Army  Identification  (AI) 

- 

.60 

.59 

RBI:  Impulsiveness  (IMP) 

- 

1.20 

1.20 

RBI:  Educational  Identification  (EDID) 

.73 

.74 

RBI:  Response  Distortion  (RD) 

- 

1.32 

1.31 

PFC:  Persistence  (PERSIST) 

- 

1.04 

1.05 

PFC:  Group  Orientation  (GRORIENT) 

- 

.92 

.91 

d  for  Model  Predicted  Value 

.14 

.98 

.99 

Note:  N  =  994-1015.  Cell  values  for  each  predictor  represent  odds  ratios.  Odds  ratios  greater  than  one  indicate 
increases  in  scores  on  the  given  predictor  are  associated  with  increases  in  the  risk  of  disenrollment.  Odds  ratios 
less  than  one  indicate  increases  in  scores  on  the  given  predictor  are  associated  with  decreases  in  the  risk  of 

disenrollment.  d  —  (iHpre(iicted  Score  for  Disenrolled  Cadets  ^Predicted  Score  for  Enrolled  CadetsVEooled  SD.  Predicted  Values  are 

scaled  in  such  a  way  that  higher  values  reflect  greater  likelihood  of  disenrollment.  Boldface  text  indicates  odds 
ratios  was  significant  ( p  <.05,  two-tailed). 

Given  the  “optimal”  nature  of  regression- weighted  composites  noted  earlier,  the 
criterion-related  validity  estimates  presented  for  the  composite  of  RBI  and  PFC  scales  in 
Model  2  in  Table  5.2  (d  =  .98)  may  be  optimistic.  Table  5.3  attempts  to  deal  with  this  issue 
by  presenting  results  for  unit-weighted  composites  of  a  reduced  set  of  RBI-PFC  scales.  Not 
surprisingly,  results  in  Table  5.3  reveal  that  a  reduced  unit-weighted  composite  of  RBI  and 
PFC  scales  appeared  to  have  lower  levels  of  criterion  related  validity  (Model  4  d  =  .64) 
relative  to  the  regression-weighted  composite  of  all  RBI-PFC  scales  (Model  2  d  =  .98). 
Nevertheless,  this  unit-weighted  composite  still  produced  good  levels  of  prediction.  For 
example,  based  on  results  for  Model  4  in  Table  5.3,  for  every  one  standard  deviation  increase 
in  unit- weighted  RBI-PFC  composite  scores,  the  odds  were  1.71  times  lower  that  a  Cadet 
would  disenroll. 
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Table  5.3.  Logistic  Regression  Results  for  Unit-Weighted  RBI+PFC  Composite  with 
Disenrollment  as  the  Criterion 


Odds  Ratios 

Predictor 

Model  1 
(WPS  Only) 

Model  4 
(Unit-Weight 
RBI+PFC) 

Model  5  (WPS  + 
Unit-Weight 
RBI+PFC) 

Whole  Person  Score  (WPS) 

.88 

- 

.93 

Unit-Weighted  RBI+PFC  Composite 

- 

.58 

.59 

d  for  Model  Predicted  V alue 

.14 

.64 

.65 

Note:  N  =  994-999.  Cell  values  for  each  predictor  represent  odds  ratios.  Odds  ratios  less  than  one  indicate 
increases  in  scores  on  the  given  predictor  are  associated  with  decreases  in  the  risk  of  disenrollment.  d  = 

(7ffpredicted  Score  for  Disenrolled  Cadets  ^^Predicted  Score  for  Enrolled  CadetsVPooled  SI).  Predicted  Values  are  Scaled  ill  SUCh  a  Way 

that  higher  values  reflect  greater  likelihood  of  disenrollment.  The  Unit-Weighted  RBI+PFC  composite  was 
constructed  by  averaging  standard  scores  for  scales  appearing  on  a  reduced  version  of  the  CBEF  that  reflected 
changes  to  the  CBEF  made  as  a  result  of  the  current  effort  (see  Chapter  6).  The  CBEF  scales  included  in  this 
composite  were:  the  RBI  SE,  ST,  ACH,  AI,  FM,  EDID,  and  HOST  scales  and  the  PFC  GRORIENT  scale. 
Boldface  text  indicates  odds  ratios  was  significant  (p<.05,  two-tailed). 


Regression  Results  for  Predicting  Cadets’  Likelihood  of  Becoming  an  Army  Officer 

Regression  results  using  Cadets’  self-rated  likelihood  of  becoming  an  Army  Officer 
as  the  focal  criterion  are  presented  in  Table  5.4.  Recall  from  Chapter  4  that  the  self-rated 
likelihood  of  becoming  an  Army  Officer  criterion  reported  here  reflects  a  dichotomized 
variable  (l=More  Likely,  0  =  Less  Likely).  Results  are  fairly  consistent  with  those  for  the 
disenrollment  criterion.  For  example,  once  again  RBI  Army  Identification  and  Educational 
Identification  appear  to  have  been  the  strongest  predictors.  For  example,  for  every  one 
standard  deviation  increase  in  RBI  Army  Identification  scores,  the  odds  were  2.86  greater 
that  a  Cadet  viewed  him/herself  as  more  likely  to  become  an  Army  Officer  (holding  all  other 
predictors,  including  the  WPS,  constant).  For  every  one  standard  deviation  increase  in  RBI 
Educational  Identification  scores,  the  odds  were  1.49  times  greater  that  a  Cadet  viewed 
him/herself  as  more  likely  to  become  an  Army  Officer  (again,  holding  all  other  predictors 
constant).  One  difference  between  the  results  presented  here  relative  to  the  results  presented 
for  disenrollment  was  that  the  RBI  Response  Distortion  scale  was  not  a  significant  predictor 
of  Cadets’  self-rated  likelihood  of  becoming  an  Army  Officer. 

As  with  the  disenrollment  criterion,  there  was  a  notable  increase  in  criterion-related 
validity  when  the  RBI  and  PFC  scales  were  added  to  a  model  containing  the  WPS  only  (e.g., 
Model  1  d  =  .04  vs.  Model  3  cl  =  1.99).  A  comparison  of  chi-squared  fit  statistics  for  these 
models  revealed  that  this  increment  in  validity  was  statistically  significant  (Ax'  [15]  = 

1 14.18,  p  <  .01,  one-tailed).  Further  examination  revealed  that  this  improvement  in  validity 
was  attributable  to  the  RBI  scales,  as  a  comparison  of  a  model  that  included  the  WPS  and 
RBI  scales  only  to  Model  3  revealed  that  addition  of  the  PFC  scales  failed  to  significantly 
increment  the  validity  of  the  reduced  model  (Ax'  [2]  =  0.65,  ns,  one-tailed). 
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Table  5.4.  Logistic  Regression  Results  with  Cadets’  Self-Rated  Likelihood  of  Becoming 
an  Army  Officer  as  the  Criterion 


Predictor 

Model  1 
(WPS  Only) 

Odds  Ratios 
Model  2 
(RBI+PFC) 

Model  3 

(WPS+RBI+PFC) 

Whole  Person  Score  (WPS) 

0.95 

- 

0.89 

RBI:  Peer  Leadership  (PLEAD) 

- 

0.90 

0.91 

RBI:  Cognitive  Flexibility  (CF) 

- 

1.03 

1.04 

RBI:  Achievement  Orientation  (ACH) 

- 

1.37 

1.37 

RBI:  Fitness  Motivation  (FM) 

- 

0.92 

0.93 

RBI:  Interpersonal  Skills  -  Diplomacy  (ISDIP) 

- 

1.01 

1.01 

RBI:  Stress  Tolerance  (ST) 

- 

1.32 

1.35 

RBI:  Hostility  to  Authority  (HOST) 

- 

1.09 

1.09 

RBI:  Self-Efficacy  (SE) 

- 

1.08 

1.08 

RBI:  Cultural  Tolerance  (CT) 

- 

0.83 

0.83 

RBI:  Army  Identification  (AI) 

- 

2.94 

2.86 

RBI:  Impulsiveness  (IMP) 

- 

0.72 

0.72 

RBI:  Educational  Identification  (EDID) 

- 

1.47 

1.49 

RBI:  Response  Distortion  (RD) 

- 

0.81 

0.80 

PFC:  Persistence  (PERSIST) 

- 

0.95 

0.96 

PFC:  Group  Orientation  (GRORIENT) 

- 

1.16 

1.16 

d  for  Model  Predicted  Value 

.04 

2.00 

1.99 

Note.  N  =  1015-1020.  Cell  values  for  each  predictor  represent  odds  ratios.  Odds  ratios  greater  than  one  indicate 
increases  in  scores  on  the  given  predictor  are  associated  with  increases  in  Cadets’  self-rated  likelihood  of 
becoming  an  Army  Officer.  Odds  ratios  less  than  one  indicate  increases  in  scores  on  the  given  predictor  are 
associated  with  decreases  Cadets’  self-rated  likelihood  of  becoming  an  Army  Officer,  d  =  (MPre(iiCted  score  for  -More 
Likely”  Cadets  —  ^Predicted  Score  for  “Less  Likely”  Cadets)/Pooled  SD.  Predicted  values  are  scaled  in  such  a  way  that  higher 
values  reflect  greater  likelihood  of  becoming  an  Army  Officer.  Boldface  text  indicates  odds  ratios  was 
significant  (/;><. 05,  two-tailed). 

As  was  the  case  with  the  disenrollment  criterion,  given  the  optimization  of  regression 
weights  for  RBI  and  PFC  scales  for  the  current  sample  of  Cadets,  the  criterion-related 
validity  estimates  presented  for  the  composite  of  RBI  and  PFC  scales  in  Model  2  in  Table  5.4 
( d  =  2.00)  may  be  optimistic.  Results  presented  in  Table  5.5  attempt  to  deal  with  this  issue  by 
presenting  results  for  unit- weighted  composites  of  a  reduced  set  of  RBI-PFC  scales.  Not 
surprisingly,  results  in  Table  5.5  reveal  that  a  reduced  unit-weighted  composite  of  RBI  and 
PFC  scales  appeared  to  have  lower  levels  of  criterion  related  validity  (Model  4  d  =  1.39) 
relative  to  the  regression- weighted  composite  of  all  RBI-PFC  scales  (Model  2  d  =  2.00). 
Nevertheless,  this  unit-weighted  composite  still  produced  good  levels  of  prediction.  For 
example,  based  on  results  for  Model  4  in  Table  5.4,  for  every  one  standard  deviation  increase 
in  unit- weighted  RBI-PFC  composite  scores,  the  odds  were  3.08  times  greater  that  a  Cadet 
viewed  him/herself  as  more  likely  to  become  an  Army  Officer. 
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Table  5.5.  Logistic  Regression  Results  for  Unit-Weighted  RBI+PFC  Composite  with 
Cadets’  Self-Rated  Likelihood  of  Becoming  an  Army  Officer  as  the  Criterion 


Predictor 

Odds  Ratios 

Model  1 
(WPS  Only) 

Model  4 
(CBEF  Unit 
Composite) 

Model  5 
(WPS+CBEF 
Unit  Composite) 

Whole  Person  Score  (WPS) 

0.95 

- 

0.81 

Unit-Weighted  RBI+PFC  Composite 

- 

3.08 

3.17 

d  for  Model  Predicted  V alue 

0.04 

1.39 

1.40 

Note.  N  =  1015-1020.  Cell  values  for  each  predictor  represent  odds  ratios.  Odds  ratios  greater  than  one  indicate 
increases  in  scores  on  the  given  predictor  are  associated  with  increases  in  Cadets’  self-rated  likelihood  of 
becoming  an  Army  Officer.  Odds  ratios  less  than  one  indicate  increases  in  scores  on  the  given  predictor  are 
associated  with  decreases  Cadets’  self-rated  likelihood  of  becoming  an  Army  Officer,  d  =  (MPredicted  Score  for  ..More 

Likely”  Cadets  —  ^Predicted  Score  for  “Less  Likely”  cadets)/Pooled  SD .  Predicted  values  are  scaled  in  such  a  way  that  higher 

values  reflect  greater  likelihood  of  becoming  an  Army  Officer.  Boldface  text  indicates  odds  ratios  was 
significant  (p<. 05,  two-tailed). 

Regression  Results  for  Predicting  Cadets  ’  Likelihood  of  Making  the  Army  a  Career 

Regression  results  using  Cadets’  self-rated  likelihood  of  making  the  Army  a  career  as 
the  focal  criterion  are  presented  in  Table  5.6.  Recall  from  Chapter  4,  that  the  self-rated 
likelihood  of  making  the  Army  a  career  criterion  reported  here  reflects  a  dichotomized 
variable  (1=  Likely,  0  =  Unlikely).  Results  for  this  criterion  differ  in  some  notable  ways 
relative  to  criteria  presented  above.  For  example,  unlike  previous  regression  results,  WPS 
exhibited  a  statistically  significant  negative  relationship  with  Cadets’  self-rated  likelihood  of 
making  the  Army  a  career.  This  relationship  emerged  in  the  case  when  WPS  scores  were 
considered  in  isolation  (Model  1)  as  well  as  when  considered  in  the  context  of  RBI  and  PFC 
scores  (Model  3).  For  example,  based  on  Model  3,  for  every  one  standard  deviation  increase 
in  WPS  scores,  there  was  a  47%  decrease  in  the  odds  that  a  Cadet  viewed  him/herself  as 
likely  to  make  the  Army  a  career  (holding  scores  on  all  RBI  and  PFC  scales  constant).  Such  a 
finding  may  be  indicative  that  Cadets  who  have  high  WPS  scores  perceive  that  they  have 
more  potential  civilian  job  opportunities  relative  to  Cadets  who  have  low  WPS  scores — and 
as  such  may  view  themselves  as  less  likely  to  make  the  Army  a  career. 

Another  difference  between  results  for  this  criterion  relative  to  results  for 
disenrollment  and  self-rated  likelihood  of  becoming  an  Army  Officer  presented  earlier  regard 
the  significance  of  RBI  scales.  Although  RBI  Army  Identification  once  again  emerged  as  a 
statistically  significant  predictor,  RBI  Educational  Identification  did  not.  Nevertheless,  RBI 
Self-Efficacy  did  emerge  as  a  statistically  significant  predictor  of  Cadets’  self-rated 
likelihood  of  making  the  Army  a  career.  Additionally,  note  that  the  magnitude  of  the  relation 
between  RBI  Army  Identification  and  likelihood  of  making  the  Army  a  career  appeared  to  be 
stronger  relative  to  its  relation  with  the  other  criteria.  For  example,  for  every  one  standard 
deviation  increase  in  RBI  Army  Identification  scores,  the  odds  were  3.45  times  greater  that  a 
Cadet  viewed  him/herself  as  likely  to  make  the  Army  a  career  (holding  all  other  predictors, 
including  the  WPS,  constant).  The  increased  contribution  of  RBI  Army  Identification  and 
decreased  contribution  of  RBI  Educational  Identification  when  predicting  self-rated 
likelihood  of  making  the  Army  a  career  relative  to  the  other  criteria  make  intuitive  sense  in 
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Table  5.6.  Logistic  Regression  Results  for  Cadets’  Self-Rated  Likelihood  of  Making  the 
Army  Career  as  the  Criterion 


Odds  Ratios 

Predictor 

Model  1 
(WPS  Only) 

Model  2 
(RBI+PFC) 

Model  3 

(WPS+RBI+PFC) 

Whole  Person  Score  (WPS) 

0.65 

- 

0.68 

RBI:  Peer  Leadership  (PLEAD) 

- 

0.81 

0.83 

RBI:  Cognitive  Flexibility  (CF) 

- 

0.78 

0.82 

RBI:  Achievement  Orientation  (ACH) 

- 

1.30 

1.30 

RBI:  Fitness  Motivation  (FM) 

- 

0.77 

0.79 

RBI:  Interpersonal  Skills  -  Diplomacy  (ISDIP) 

- 

0.88 

0.88 

RBI:  Stress  Tolerance  (ST) 

- 

0.88 

0.93 

RBI:  Hostility  to  Authority  (HOST) 

- 

0.91 

0.92 

RBI:  Self-Efficacy  (SE) 

- 

1.61 

1.59 

RBI:  Cultural  Tolerance  (CT) 

- 

1.19 

1.19 

RBI:  Army  Identification  (AI) 

- 

3.57 

3.45 

RBI:  Impulsiveness  (IMP) 

- 

1.09 

1.06 

RBI:  Educational  Identification  (EDID) 

- 

0.81 

0.86 

RBI:  Response  Distortion  (RD) 

- 

1.28 

1.22 

PFC:  Persistence  (PERSIST) 

- 

0.92 

0.93 

PFC:  Group  Orientation  (GRORIENT) 

- 

1.08 

1.03 

d  for  Model  Predicted  Value 

.44 

1.76 

1.83 

Note.  N  =  1015-1020.  Cell  values  for  each  predictor  represent  odds  ratios.  Odds  ratios  greater  than  one  indicate 
increases  in  scores  on  the  given  predictor  are  associated  with  increases  in  Cadets’  self-rated  likelihood  of 
making  the  Army  a  career.  Odds  ratios  less  than  one  indicate  increases  in  scores  on  the  given  predictor  are 
associated  with  decreases  Cadets’  self-rated  likelihood  of  making  the  Army  a  career,  d  =  (MPredicted  Score  for  •.Likely» 
Cadets  —  Mpredicted  Score  for  “Unikeiy”  cadets)/Pooled  SD.  Predicted  values  are  scaled  in  such  a  way  that  higher  values 
reflect  greater  likelihood  of  making  the  Army  a  career.  Boldface  text  indicates  odds  ratios  was  significant 
(/^<.05,  two-tailed). 

that  the  former  should  be  much  more  a  function  of  a  Cadet’s  identification  with  the  Army 
rather  than  his/her  desire  to  obtain  a  college  degree. 

As  with  the  other  criteria,  there  was  a  notable  increase  in  criterion-related  validity 
when  the  RBI  and  PFC  scales  were  added  to  a  model  containing  the  WPS  only  (e.g.,  Model  1 
cl  =  .44  vs.  Model  3  d  =  1.83).  A  comparison  of  chi-squared  model  fit  statistics  for  these 
models  revealed  that  this  increment  in  validity  was  statistically  significant  (Ax'  [15]  = 

154.34,  p  <  .01,  one-tailed).  Further  examination  revealed  that  this  improvement  in  validity 
was  attributable  to  the  RBI  scales,  as  comparison  of  a  model  that  included  the  WPS  and  RBI 
scales  only  to  Model  3  revealed  that  addition  of  the  PFC  scales  failed  to  significantly 
increment  the  validity  of  the  reduced  model  (Ax~  [2]  =  0.16,  ns,  one-tailed). 

As  was  the  case  with  the  other  criteria,  given  the  optimization  of  regression  weights 
for  RBI  and  PFC  scales  for  the  current  sample  of  Cadets,  the  criterion-related  validity 
estimates  presented  for  the  composite  of  RBI  and  PFC  scales  in  Model  2  in  Table  5.6  ( d  = 
1.76)  may  be  optimistic.  Results  presented  in  Table  5.7  attempt  to  deal  with  this  issue  by 
presenting  results  for  unit- weighted  composites  of  a  reduced  set  of  RBI-PFC  scales.  As  with 
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Table  5. 7.  Logistic  Regression  Results  for  Unit-Weighted  RBI+CBEF  Composite  with 
Cadets’  Self-Rated  Likelihood  of  Making  the  Army  Career  as  the  Criterion 


Predictor 

Odds  Ratios 

Model  1 
(WPS  Only) 

Model  4 
(Unit- Weight 
RBI+PFC ) 

Model  5  (WPS  + 
Unit- Weight 
RBI+PFC ) 

Whole  Person  Score  (WPS) 

0.65 

- 

0.56 

Unit-Weighted  RBI+PFC  Composite 

- 

2.26 

2.48 

d  for  Model  Predicted  V alue 

0.44 

0.90 

1.07 

Note.  N  =  1015-1020.  Cell  values  for  each  predictor  represent  odds  ratios.  Odds  ratios  greater  than  one  indicate 
increases  in  scores  on  the  given  predictor  are  associated  with  increases  in  Cadets’  self-rated  likelihood  of 
making  the  Army  a  career.  Odds  ratios  less  than  one  indicate  increases  in  scores  on  the  given  predictor  are 
associated  with  decreases  Cadets’  self-rated  likelihood  of  making  the  Army  a  career,  d  =  (MPredicted  Score  for  •.Likely- 
cadets  —  ^Predicted  score  for  ‘-unikeiy”  cadets)/Pooled  SD.  Predicted  values  are  scaled  in  such  a  way  that  higher  values 
reflect  greater  likelihood  of  making  the  Army  a  career.  Boldface  text  indicates  odds  ratios  was  significant 
(p<. 05,  two-tailed). 

the  other  criteria,  results  in  Table  5.7  reveal  that  a  reduced  unit-weighted  composite  of  RBI 
and  PFC  scales  appeared  to  have  lower  levels  of  criterion  related  validity  (Model  4  d  =  0.90) 
relative  to  the  regression- weighted  composite  of  all  RBI-PFC  scales  (Model  2  d  =  1.76). 
Nevertheless,  this  unit-weighted  composite  still  produced  good  levels  of  prediction.  For 
example,  based  on  results  for  Model  4  in  Table  5.7,  for  every  one  standard  deviation  increase 
in  unit-weighted  RBI-PFC  composite  scores,  the  odds  were  2.26  times  greater  that  a  Cadet 
viewed  him/herself  as  likely  to  make  the  Army  a  career. 

Potential  Impact  of  Considering  the  CBEF  in  the  Scholarship  Awarding  Process 

The  above  results  indicate  that  the  CBEF  offers  considerable  promise  in 
supplementing  the  WPS  to  reduce  disenrollment  of  four-year  scholarship  recipients.  To 
establish  a  better  sense  of  the  value  that  the  CBEF  may  add  to  WPS  when  awarding 
scholarships.  Table  5.8  presents  disenrollment  rates  for  Cadets  who  scored  in  different  ranges 
of  several  potential  predictor  composites.  These  composites  reflect  rescaled  predicted  values 
from  the  five  models  presented  earlier  in  the  section  on  regression  results  for  disenrollment. 
Specifically,  results  in  Table  5.8  are  based  on  composite  scores  that  have  been  scaled  in  a 
way  such  that  higher  scores  reflect  a  lower  expected  likelihood  of  disenrollment. 

To  help  interpret  the  results  in  Table  5.8,  it  is  useful  to  consider  that  the  base  rate  of 
disenrollment  among  Cadets  who  completed  the  CBEF  was  10.3%.  Taking  results  for  Model 
4  as  an  example,  Table  5.8  shows  that  Cadets  who  scored  in  the  top  75%  of  a  composite 
formed  from  a  unit-weighted  combination  of  RBI  and  PFC  scale  scores  (again,  scaled  in  a 
way  such  that  higher  scores  reflect  a  lower  expected  likelihood  of  disenrollment)  had 
disenrollment  rates  ranging  from  4.45  to  9.16%,  whereas  Cadets  who  scored  in  the  bottom 
25%  of  that  composite  had  disenrollment  rates  of  20.8% — double  the  disenrollment  rate  of 
Cadets  in  general  in  the  current  sample.  Contrast  this  finding  with  the  disenrollment  rate  of 
Cadets  on  the  bottom  25%  of  the  WPS  score  distribution  (13.39%)  and  the  difference  in  the 
relative  risk  of  Cadets  at  the  lower  end  of  these  distributions  becomes  apparent.  Such 
findings,  coupled  with  the  criterion-related  validity  results  presented  earlier  suggest  that  the 
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CBEF  may  prove  effective  as  a  measure  to  screen  out  a  given  percentage  of  applicants  who 
would  be  at  heightened  risk  for  disenrollment  relative  to  the  Cadet  population  in  general. 


Table  5.8.  Disenrollment  Rates  by  Quartile  of  Rescaled  Model  Predicted  Values 


Model 

Quartile 

Enrolled 

Disenrolled 

Total 

Disenrollment 

Rate 

1 

Bottom  25% 

220 

34 

254 

13.39 

WPS  Only 

26-50% 

214 

18 

232 

7.76 

51-75% 

225 

20 

245 

8.16 

Top  25% 

225 

27 

252 

10.71 

2 

Bottom  25% 

194 

51 

245 

20.82 

Regression 

26-50% 

226 

23 

249 

9.24 

RBI+PFC 

51-75% 

239 

13 

252 

5.16 

Top  25% 

232 

15 

247 

6.07 

3 

Bottom  25% 

191 

54 

245 

22.04 

Regression 

26-50% 

228 

21 

249 

8.43 

WPS+RBI+PFC 

51-75% 

237 

14 

251 

5.58 

Top  25% 

235 

13 

248 

5.24 

4 

Bottom  25% 

198 

52 

250 

20.80 

Unit-Weighted 

26-50% 

236 

11 

247 

4.45 

RBI+PFC 

51-75% 

228 

23 

251 

9.16 

Top  25% 

235 

16 

251 

6.37 

5 

Bottom  25% 

200 

50 

250 

20.00 

WPS  + 

26-50% 

230 

15 

245 

6.12 

Unit- Weighted 

51-75% 

232 

19 

251 

7.57 

RBI+PFC 

Top  25% 

235 

18 

253 

7.11 

Note.  Predicted  values  are  scaled  in  such  a  way  that  higher  values  reflect  a  lower  expected  likelihood  of 
disenrollment.  Model  1=  WPS  only.  Model  2  =  Regression-weighted  RBI-PFC  scale  composite.  Model  3  = 
Regression-weighted  WPS-RBI-PFC  scale  composite.  Model  4  =  Unit-weighted  RBI-PFC  composite.  Model  5 
=  Unit-weighted  WPS-RBI-PFC  composite. 
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Chapter  6:  Next  Steps  for  the  CBEF 

Teresa  Russell 
HumRRO 

Trueman  Tremble 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

As  described  at  the  beginning  of  this  report,  the  focus  of  the  current  effort  was  to 
develop  a  new  measure  that  improves  the  prediction  of  ROTC  continuance  for  four- year 
scholarship  recipients,  beyond  the  level  of  prediction  already  afforded  by  the  current 
scholarship  award  process,  specifically  Whole  Person  Scores.  Chapters  2-5  of  this  report 
presented  data  showing  that  the  CBEF  has  a  great  deal  of  promise  for  identifying  Cadets  who 
are  at  heightened  risk  for  disenrollment.  ARI  and  HumRRO  briefed  Cadet  Command  on 
these  results  in  January  of  2008. 

Nevertheless,  the  results  are  just  a  first  step  in  a  larger  research  process  examining  the 
potential  for  using  the  CBEF  in  the  process  of  awarding  four-year  scholarships.  In  this 
chapter,  we  provide  an  overview  of  the  larger  program  of  research  surrounding  the  CBEF, 
and  describe  plans  for  its  future  evaluation. 

Revisions  to  the  CBEF 

We  revised  the  content  of  the  CBEF  based  on  a  review  of  the  psychometric  properties 
of  the  version  of  the  CBEF  examined  in  the  earlier  chapters.  ARI  personnel,  HumRRO  staff, 
and  an  external  consultant  with  expertise  in  the  development  and  validation  of  selection 
measures  met  on  several  occasions  to  (a)  identify  CBEF  scales  that  should  be  dropped  or 
improved;  (b)  identify  scales  that  should  be  added  to  the  CBEF;  (c)  write  new  items  as 
needed;  and  (d)  ensure  that  all  items  were  appropriate  for  high  school  students  with  no 
military  experience. 

Based  on  those  discussions,  we  dropped  the  following  CBEF  scales:  Cultural 
Tolerance,  Cognitive  Flexibility,  Impulsiveness,  Interpersonal  Skills,  and  Peer  Feadership. 
Additional  Educational  Identification  biodata  items  were  written  to  improve  the  internal 
consistency  of  this  scale  (recall  its  relatively  low  reliability  presented  in  Table  4.1,  a  =  .49). 
Finally,  we  added  several  new  RBI  scales:  Attitudes  Toward  Deployment,  Continuance 
Commitment,  Past  Withdrawal  Propensity,  and  Traditional  Values. 

We  also  edited  the  PFC  items  to  improve  their  readability  and  did  some  follow-up 
analyses  on  the  PFC  to  make  further  refinements.  As  described  in  Chapter  2,  15  factors  were 
extracted  in  an  initial  factor  analysis  of  PFC  items  for  the  current  sample.  Three  researchers 
met  to  discuss  possible  reasons  and  solutions  for  this  heterogeneity.  Researchers  concluded 
that  (a)  many  of  the  descriptors  anchoring  the  two  ends  of  each  item  were  confusing  or 
unnecessarily  complex,  (b)  the  two  descriptors  in  several  items  were  likely  measuring  very 
different  things,  and  (c)  too  many  assumptions  were  required  when  answering  the  items. 
Therefore,  we  revised  the  items.  The  descriptors  were  shortened  considerably  by  creating  a 


41 


stem  for  each  item.  Thus,  only  a  few  words,  at  most,  were  needed  for  the  two  (i.e.,  bottom 
and  top)  descriptors.  Several  items  were  dropped  because  they  could  not  be  revised  to  fit  the 
new  form  (e.g.,  the  two  descriptors  described  different  situations).  The  researchers  also 
refined  the  PFC  scales  and  sorted  items  into  the  new  scales.  The  revised  PFC  comprises  18 
items  divided  across  three  scales: 

•  Group  Orientation 

•  Propensity  for  Group  Commitment 

•  Need  for  Autonomy  vs.  Need  for  Affiliation 

The  Group  Orientation  and  Propensity  for  Group  Commitment  scale  are  conceptually  similar 
to  the  PFC  Group  Orientation  and  PFC  Persistence  scales  analyzed  in  this  report.  The  new 
Need  for  Autonomy  vs.  Need  for  Affiliation  scale  represents  a  subset  of  original  items 
augmented  by  new  items  that  attempt  to  isolate  a  specific  preference  for  working  alone 
versus  as  part  of  a  group. 

Finally,  we  added  a  “work  values”  section  asking  respondents  to  rank  order  nine  work 
characteristics  in  terms  of  their  relative  importance  on  their  “ideal  job”.  The  characteristics 
chosen  were  carefully  selected  to  reflect  a  balance  of  work  characteristics  that  would 
generally  either  be  (a)  descriptive  of  the  work  environment  of  Army  Officers  or  (b)  not 
descriptive  of  the  work  environment  of  Army  Officers.  The  rationale  behind  such  a  measure 
is  that  Cadets  who  place  more  importance  on  work  characteristics  descriptive  of  the  work 
environment  of  Army  Officers  (relative  to  those  that  are  not  descriptive)  would  be  more 
likely  to  experience  positive  outcomes  (e.g.,  actually  becoming  an  Army  Officer,  making  the 
Army  a  career)  compared  to  Cadets  who  place  more  importance  on  work  characteristics  that 
are  not  descriptive  of  the  work  environment  of  Army  Officers.  An  overview  of  the  final 
content  of  the  CBEF-2  appears  in  Table  6.1. 

Development  of  the  Cadet  Experience  Form  (CEF) 

In  anticipation  of  future  evaluations  of  the  CBEF,  we  developed  a  self-report  criterion 
measure  as  part  of  the  current  effort — the  Cadet  Experience  Form  (CEF).  The  CEF  asks 
respondents  to  describe  their  attitudes  toward  college,  ROTC,  and  the  Army.  It  is  appropriate 
for  administration  to  Cadets  who  have  some  experience  with  college,  ROTC,  and  the  Army; 
that  is.  Cadets  who  are  in  the  spring  of  their  freshmen  year  or  later.  To  develop  this  measure, 
we  drew  on  our  experience  in  previous  Army  research  using  similar  measures  (Strickland, 
2004)  and  also  adapted  items  from  the  Cadet  Command’s  On  Campus  Market  Potential 
(OCMP)  survey  which  is  periodically  administered  to  college  students  on  campuses  that  host 
Army  ROTC.  While  the  specific  items  on  the  OCMP  survey  have  varied  over  the  years,  the 
surveys  typically  contain  a  core  of  similar  items  about  students’  attitudes  toward  the  ROTC 
program. 


Additional  Data  Collection  Efforts 

This  report  studied  the  validity  of  the  original  CBEF  by  longitudinally  tracking  a 
sample  of  freshmen  four- year  ROTC  scholarship  recipients  tested  in  SY06-07.  Since  then, 
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plans  have  been  made  to  administer  the  updated  CBEF  (CBEF-2)  to  other  ROTC  Cadet 
cohorts,  especially,  students  earlier  in  their  affiliation  with  ROTC  (e.g.,  high  school  students 
who  are  applying  for  a  scholarship).  Future  longitudinal  research  will  examine  the  validity  of 
the  CBEF  for  predicting:  (a)  actual  entry  into  the  ROTC  program,  (b)  full  program 
completion  and  commissioning  into  the  Army,  and  (c)  career  continuance  in  the  Army 
beyond  the  service  obligation  incurred  with  ROTC. 

Table  6.1.  Overview  of  Content  on  the  CBEF-2 

RBI:  Achievement  Orientation:  The  willingness  to  give  one’s  best  effort  and  to  work  hard  towards  achieving 
difficult  objectives. 

RBI:  Army  Identification  The  degree  of  personal  identification  with,  and  intrinsic  interest  in  becoming,  a  U.S. 
Army  Officer. 

RBI:  Attitudes  Toward  Deployment:  The  tendency  to  participate  in  activities  associated  with  risk  of  self- 
injury  and  danger. 

RBI:  Continuance  Commitment:  Degree  of  commitment  to  finishing  college  or  completing  ROTC  because 
(a)  the  cost  of  leaving  is  too  high,  (b)  the  quality  of  his/her  options  will  be  better,  or  (c)  it  is  a  way  to  achieve 
his/her  ultimate  career  goal. 

RBI:  Educational  Identification:  Seeks  a  college  degree.  Perceives  self  as  someone  who  will  complete 
college. 

RBI:  Fitness  Motivation:  Degree  of  enjoyment  from  participating  in  physical  exercise.  Willingness  to  put  in 
the  time  and  effort  to  maintain  good  physical  conditioning. 

RBI:  Hostility  to  Authority:  Being  suspicious  of  the  motives  and  actions  of  legitimate  authority  figures. 
Viewing  rules,  regulations,  and  directives  from  higher  authority  as  punitive  and  illegitimate. 

RBI:  Response  Distortion:  The  purpose  is  to  detect  and  adjust  for  socially  desirable  responding. 

RBI:  Past  Withdrawal  Propensity:  The  tendency  to  withdraw  from  commitments  (e.g.,  high  school,  jobs). 

RBI:  Self-Efficacy:  Feeling  that  one  has  successfully  overcome  work  obstacles  in  the  past  and  that  one  will 
continue  to  do  so  in  the  future. 

RBI:  Stress  Tolerance:  Ability  to  maintain  one’s  composure  under  pressure.  Remaining  calm  and  in  control  of 
one’s  emotions  instead  of  feeling  anxious  and  worried. 

RBI:  Traditional  Values:  Acceptance  of  societal  values,  authority,  and  the  value  of  discipline. 

PFC:  Group  Orientation:  The  tendency  to  seek  out  and  affiliate  with  a  group. 

PFC:  Propensity  for  Group  Commitment:  The  tendency  to  persist  with  one’s  group  membership,  even  when 
problems  arise. 

PFC:  Need  for  Autonomy  vs.  Need  for  Affiliation:  The  tendency  to  prefer  working  alone  versus  in  a  group. 

Work  Values  Profile  Consistent/Inconsistent  with  the  Work  of  an  Army  Officer 


Summary 

As  described  at  the  beginning  of  this  report,  the  focus  of  the  current  effort  was  to 
develop  a  new  measure  that  improves  the  prediction  of  ROTC  continuance  for  four- year 
scholarship  recipients.  Chapters  2-5  of  this  report  described  how  three  criterion  variables 
were  used  to  evaluate  the  potential  of  the  CBEF’s  RBI  and  PFC  scales  for  predicting  (a) 
disenrollment  (versus  enrollment),  (b)  Cadets’  self-reported  likelihood  of  becoming  an  Army 
Officer,  and  (c)  Cadets’  self-reported  likelihood  of  making  the  Army  a  career.  When 
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considered  in  isolation,  several  CBEF  scales  were  significantly  predictive  of  all  three  criteria, 
notably  RBI  Army  Identification,  Achievement  Orientation,  Fitness  Motivation,  Self- 
Efficacy,  Hostility  to  Authority,  and  PFC  Persistence  (Table  5.1).  When  considered  in  the 
context  of  all  CBEF  scales  and  the  WPS  via  logistic  regression  analyses:  (a)  only  the  RBI 
Army  Identification  scale  remained  significantly  related  to  all  three  criteria,  (b)  the  RBI 
Educational  Identification  scale  was  significantly  related  to  disenrollment  and  Cadets’  self- 
rated  likelihood  of  becoming  an  Army  Officer,  and  (c)  the  RBI  Self-Efficacy  scale  was 
significantly  related  to  Cadets’  self-rated  likelihood  of  making  the  Army  a  career  (Tables  5.2 
through  5.7).  Furthermore,  these  logistic  regression  analyses  indicated  that  addition  of  CBEF 
scale  scores  (in  particular,  RBI  scale  scores)  to  a  model  including  only  WPS  significantly 
incremented  the  validity  of  WPS  for  predicting  all  three  criteria  (Tables  5.2  through  5.7). 

Implications  of  Findings 

The  results  summarized  in  this  report  suggest  there  is  good  potential  for  developing 
selection  measures  designed  to  predict  disenrollment  among  four- year  high  school 
scholarship  applicants.  These  findings  encourage  further  research  on  a  selection  measure  for 
use  in  an  operational  setting  (i.e.,  as  part  of  the  actual  process  for  awarding  four-year  high 
school  scholarships).  For  example,  the  present  study  involved  administering  the  CBEF  to 
Cadets  in  the  spring  of  their  freshman  year  in  college.  These  data  enabled  examination  of 
relationships  between  CBEF  scores  and  disenrollment  that  occurred  between  the  spring  of 
Cadets’  freshman  year  and  fall  of  their  sophomore  year,  to  include  estimations  for  a  refined 
version  of  the  CBEF  likely  used  in  subsequent  efforts.  However,  the  data  did  not  allow  for 
examination  of  disenrollment  that  occurs  (a)  prior  to  the  spring  of  Cadets’  freshman  year  or 
(b)  beyond  the  fall  of  Cadets’  sophomore  year.  Gathering  CBEF  data  from  applicants  would 
allow  one  to  examine  early  disenrollment  and  clarify  the  susceptibility  of  measures  like  the 
CBEF  to  factors  present  in  an  operational  setting  (e.g.,  strong  motivation  to  perform  well  on 
the  assessment).  As  alluded  to  above,  future  research  is  also  needed  to  determine  whether  the 
CBEF  predicts  outcomes  that  occur  beyond  Cadet’s  sophomore  year  in  college,  to  include 
graduation  from  ROTC,  performance  in  ROTC,  and  performance  as  an  Army  Officer.  In 
sum,  this  study  provides  a  promising  foundation  for  future  work  geared  towards  developing 
and  implementing  operationally  useful  measures  for  selecting  individual  who  would  be 
successful  as  ROTC  Cadets  and  later  as  a  commissioned  Army  Officer. 
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Appendix:  Data  Collection  Preparation  and  Session  Instructions 


1.  Review  and  annotate  the  roster  of  freshman  four-year  scholarship  recipients.  For  each  person  listed, 
indicate  his/her  ROTC  affiliation  status  by  checking  one  of  the  following: 

In  Program:  The  individual  is  a  MS  1  Cadet  about  to  complete  the  freshman  year. 

Dropped  Out:  The  individual  enrolled  in  MS  1  but  has  since  dropped  out  of  the 

program  (for  any  reason). 

Never  Started:  While  awarded  a  scholarship,  the  individual  did  not  enroll  in  MS  1. 

Other:  Describe  in  the  section  provided  for  notes. 

2.  Update  the  roster  for  any  Cadet  who  is  not  listed  and  who  is  a  freshman  four-year  scholarship 
recipient. 

3.  Become  familiar  with  the  purpose  of  the  study  and  the  instructions  for  a  questionnaire 
administration  session. 

4.  Arrange  to  collect  data  from  all  Cadets  on  the  updated  roster  at  the  same  time  during  one 
classroom  session.  This  session  is  expected  to  last  no  longer  than  one  50-minute  class. 


Figure  A.l.  Data  collection  preparation  instructions. 
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1 .  Distribute  the  envelopes  to  the  Cadets. 

2.  Inform  the  Cadets  of  the  study’s  purpose.  Do  this  by  reading  or  paraphrasing  the  study’s  purpose  from 
the  Cadet  Command’s  letter  of  instructions  to  the  PMS. 

3.  Have  the  Cadets  open  their  envelopes  and  check  the  contents  —  Each  envelope  should  contain  a 
privacy  act  statement  and  a  questionnaire  booklet. 

4.  Have  the  Cadet  read  the  privacy  act  statement.  As  they  do  so,  point  out  that  — 

Participation  is  voluntary.  While  making  this  point,  also  mention  that  Cadet 
participation  is  needed  for  evaluating  the  utility  of  the  questionnaires. 

The  study  asks  for  identifiers  (SSNs)  for  linking  questionnaire  responses  to  Cadet 
Command  data,  in  particular,  data  on  success  and  retention  in  ROTC. 

Provisions  are  being  taken  to  protect  confidentiality  to  include: 

Completed  materials  placed  back  in  their  original  envelope. 

Immediate  shipment  of  completed  envelops  to  the  study  team  via  Federal  Express. 

Access  to  the  data  only  by  members  of  the  study  team. 

No  report  of  findings  that  links  specific  responses  to  specific  individuals. 

5.  Answer  any  question  that  a  Cadet  may  have. 

6.  Ask  participating  Cadets  to  complete  the  privacy  act  statement  —  Signing  or  printing  their  name,  SSN, 
and  current  date. 

7.  Instruct  the  Cadets  that  the  questionnaire  has  three  parts  and  that  they  need  to  read  the  directions  for 
a  part  before  starting  it. 

8.  Then  tell  the  Cadets  to  begin  Part  I,  to  complete  the  whole  questionnaire  at  their  own  pace,  and  to  mark 
their  responses  on  the  questionnaire  book  (pen  or  pencil). 

9.  Have  Cadets  place  complete  privacy  act  statements  and  questionnaires  into  the  original  envelopes  and 
seal  the  envelopes.  Visibly  place  completed  Cadet  envelopes  into  the  return  Federal  Express  envelope. 

10.  Ship  Cadet  envelopes  and  the  roster  to  ARI  using  the  Federal  Express  materials  provided. 


Figure  A.2.  Instructions  for  data  collection  sessions. 
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